Papers with evaluation metrics
Creative Planning with Language Models: Practice, Evaluation and Applications (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | This tutorial explores how planning has been learned and deployed in creative workflows . many human creative tasks involve extensive planning, and actions need to be taken . |
| Approach: | This tutorial explores how planning has been learned and deployed in creative workflows . authors discuss forward and backward learning approaches for planning in LLMs - and evaluation metrics tailored to latent plans . |
| Outcome: | This tutorial examines how planning has been learned and deployed in creative workflows . it discusses forward and backward learning approaches for planning in LLMs - evaluation metrics tailored to latent plans . |
Summarization of Dialogues and Conversations At Scale (2023.eacl-tutorials)
Copied to clipboard
| Challenge: | Conversations are the natural communication format for people. |
| Approach: | This tutorial will survey the cutting-edge methods for summarizing written and spoken conversation. |
| Outcome: | This tutorial will examine the cutting-edge methods for summarizing written and spoken conversations, covering key sub-areas whose combination is needed for a successful solution. |
They Exist! Introducing Plural Mentions to Coreference Resolution and Entity Linking (C18-1)
Copied to clipboard
| Challenge: | Unlike singular mentions each of which represents one entity, plural mentions stand for multiple entities. |
| Approach: | They propose a novel coreference resolution algorithm that selectively creates clusters to handle both singular and plural mentions and a deep learning-based entity linking model that jointly handles both types of mentions through multi-task learning. |
| Outcome: | The proposed model outperforms existing models designed for singular mentions and plural mentions. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Shironaam: Bengali News Headline Generation using Auxiliary Information (2023.eacl-main)
Copied to clipboard
| Challenge: | Automated headline generation systems have the potential to assist editors in finding interesting headlines to attract visitors or readers. |
| Approach: | They propose to use Bengali news article-headline pairings with auxiliary data to better model headline generation using pre-trained language models. |
| Outcome: | The proposed model improves on a Bengali news headline generation dataset by 3 to 10 percentage points over baselines. |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |
Generative Induction of Dialogue Task Schemas with Streaming Refinement and Simulated Interactions (2026.tacl-1)
Copied to clipboard
| Challenge: | Slot Schema Induction (SSI) is a task-oriented dialogue (TOD) system that allows for automatic identification of information slots from unlabeled data. |
| Approach: | They propose a language model that incrementally constructs and refines a slot schema over a stream of dialogue data and then automatically creates high-quality state labels. |
| Outcome: | The proposed method creates high-quality state labels for novel task domains and improves evaluation metrics. |
BLEU might be Guilty but References are not Innocent (2020.emnlp-main)
Copied to clipboard
| Challenge: | Using a method to collect references and compare their value with human evaluations, we show that multi-reference BLEU does not improve the correlation for high quality output. |
| Approach: | They propose a method to compare the quality of automated metrics by analyzing references and comparing them with human evaluations. |
| Outcome: | The proposed method improves correlation with all modern evaluation metrics including embedding-based methods. |
EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a recent surge of interest in developing evaluation metrics based on pretrained large language models (LLMs) can better cope with lexical variation. |
| Approach: | They propose to replace computation-intensive transformers with lighter alternatives and employ linear and quadratic approximations for alignment algorithms on top of LLM representations. |
| Outcome: | The proposed approach replaces computation-intensive transformers with lighter alternatives and employs linear and quadratic approximations for alignment algorithms on top of LLM representations. |
[CASPI] Causal-aware Safe Policy Improvement for Task-oriented Dialogue (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in off-policy reinforcement learning methods that use offline data as against a simulator have proven to be sample efficient. |
| Approach: | They propose a batch-RL framework for ToD policy learning: Causal-aware Safe Policy Improvement (CASPI) that uses a mechanism to learn fine-grained reward that captures intention behind human response and offers guarantee on dialogue policy’s performance against a baseline. |
| Outcome: | The proposed framework outperforms the current state of the art on an end-to-end dialogue task using a multiwoz2.0 dataset. |
Open-Domain Question Answering (2020.acl-tutorials)
Copied to clipboard
| Challenge: | tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering (QA) |
| Approach: | tutorial provides a comprehensive overview of cutting-edge research in open-domain question answering . focus will shift to cutting- edge models proposed for open- domain QA . |
| Outcome: | The tutorial will cover cutting-edge research in open-domain question answering (QA) it will cover two-stage retriever-reader approaches, dense retriever and end-to-end training, and retriever free methods . |
A Partially Rule-Based Approach to AMR Generation (N19-3)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a representation of a sentence as a labeled graph . because of these abstractions, it can be difficult to generate from AMR back to a fluent English sentence . |
| Approach: | They propose a new approach to generating English text from Abstract Meaning Representation (AMR) it is largely rule-based, supplemented by a language model and simple statistical linearization models . they also address difficulties of automatically evaluating AMR generation systems . |
| Outcome: | The proposed approach produces a fluent English sentence with a high quality . it is difficult to generate from an AMR back to a sentence which preserves original meaning . |
GRUEN for Evaluating Linguistic Quality of Generated Text (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics focus on content selection, not linguistic quality . proposed GRUEN measures Grammaticality, non-redundancy, focUs, structure and coherence of generated text. |
| Approach: | They propose to use a BERT-based model and a class of syntactic, semantic, and contextual features to examine the system output. |
| Outcome: | Experiments show that the proposed metric correlates highly with human judgments. |
CLEME2.0: Towards Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction (2025.acl-long)
Copied to clipboard
Jingheng Ye, Zishan Xu, Yinghui Li, Linlin Song, Qingyu Zhou, Hai-Tao Zheng, Ying Shen, Wenhao Jiang, Hong-Gee Kim, Ruitong Liu, Xin Su, Zifei Shan
| Challenge: | Existing studies have focused on the interpretability of Grammatical Error Correction (GEC) evaluation metrics, but the interpretabilty of these metrics has been neglected. |
| Approach: | They propose a reference-based metric that describes four aspects of GEC systems: hit-correction, wrong-corrections, under-correcties, and over-corrects. |
| Outcome: | The proposed metric reveals critical qualities and locates drawbacks of GEC systems. |
DP-NMT: Scalable Differentially Private Machine Translation (2024.eacl-demo)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) is a popular text generation task, yet there is nagging data privacy concerns. |
| Approach: | They propose an open-source framework for a privacy-preserving NMT with DP-SGD. |
| Outcome: | The proposed framework is open-source and open to the public . it combines models, datasets, and evaluation metrics to demonstrate its effectiveness. |
PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications (2025.emnlp-industry)
Copied to clipboard
Hitesh Laxmichand Patel, Amit Agarwal, Srikant Panda, Hansa Meghwani, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan Roth
| Challenge: | Existing evaluation metrics for Multimodal Large Language Models (MLLMs) are inadequate to assess their robustness to irrelevant or distracting visual context. |
| Approach: | They propose a patch-context-robustness index to measure MLLMs' robustness to visual context variations. |
| Outcome: | The proposed score measures the robustness of MLLMs to visual contexts across 15 vision-language benchmarks. |
Vis-Eval Metric Viewer: A Visualisation Tool for Inspecting and Evaluating Metric Scores of Machine Translation Output (N18-5)
Copied to clipboard
| Challenge: | Many metrics have been proposed for Machine Translation (MT) that compare system translations against human references. |
| Approach: | They propose to use BLEU and METEOR to evaluate machine translations against human translations. |
| Outcome: | VisEval Metric Viewer (VEMV) provides visualisation of multiple evaluation scores so they can be easily interpreted by a user. |
Exploring Cross-Lingual Voice Conversion Methods for Anonymizing Low-Resource Text-to-Speech (2026.eacl-short)
Copied to clipboard
| Challenge: | a growing number of speech synthesis systems clone a person's voice, a new study finds . a variety of voice conversion techniques can mask speaker identities in low-resource text-to-speech systems. |
| Approach: | They compare voice conversion techniques to mask speaker identities in text-to-speech systems . they build and evaluate speaker-anonymized systems for two Canadian Indigenous languages . |
| Outcome: | The proposed methods are compared with other approaches for using voice conversion to mask speaker identities in low-resource text-to-speech systems. |
SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization (2021.acl-demo)
Copied to clipboard
| Challenge: | despite advances in abstractive text summarization, the true performance and failure modes of modern neural models are not yet fully understood due to the black-box nature of neural models and unmanageable scale of recent datasets for manual analysis. |
| Approach: | They propose an open-source tool for visualizing abstractive summaries that enables fine-grained analysis of models, data, and evaluation metrics associated with text summarization. |
| Outcome: | The proposed tool can identify the shortcomings and failure modes of state-of-the-art summarization models. |
AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents (2024.naacl-demo)
Copied to clipboard
Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, Carolin Lawrence
| Challenge: | Existing benchmarks are narrow and simply compute overall task success. |
| Approach: | They propose a framework where both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs. |
| Outcome: | The proposed framework can track agent progress on two use cases and identify common failure points and refine the agent architecture to obtain a significant performance increase. |
Towards Personalised and Document-level Machine Translation of Dialogue (2021.eacl-srw)
Copied to clipboard
| Challenge: | State-of-the-art (SOTA) neural machine translation systems translate texts at sentence level, ignoring context. |
| Approach: | They propose to integrate extra-textual information into the translation process for the domain of dialogue extracted from TV subtitles in five languages: English, Brazilian Portuguese, German, French and Polish. |
| Outcome: | The proposed systems translate texts at sentence level, ignoring context . there are no readily available robust evaluation metrics for them . |
Predictive Multiplicity of Knowledge Graph Embeddings in Link Prediction (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge graph embeddings (KGE) models are often used to predict missing links for knowledge graphs (KGs) however, multiple KG embedds can give conflicting predictions for unseen queries. |
| Approach: | They define predictive multiplicity in link prediction and introduce evaluation metrics to measure it using commonly used benchmark datasets. |
| Outcome: | The proposed methods significantly mitigat conflicts by 66% to 78% in link prediction. |
Generating Visual Stories with Grounded and Coreferent Characters (2026.tacl-1)
Copied to clipboard
| Challenge: | Experimental results show that our model generates visual stories with consistent and coreferent character mentions compared to baselines and state-of-the-art systems. |
| Approach: | They propose a character-centric approach to visual story generation that uses visual and textual character coreference chains to enrich the VIST benchmark. |
| Outcome: | The proposed model generates visual stories with consistent and coreferent character mentions compared to baselines and state-of-the-art systems. |
TL;DR Progress: Multi-faceted Literature Exploration in Text Summarization (2024.eacl-demo)
Copied to clipboard
| Challenge: | TL;DR Progress is a literature explorer designed specifically for the text summarization literature. |
| Approach: | They propose to organize 514 papers based on a comprehensive annotation scheme for text summarization approaches and a fine-grained, faceted search. |
| Outcome: | The proposed tool organizes 514papers based on a comprehensive annotation scheme for text summarization approaches and enables fine-grained, faceted search. |
Improving Chinese Story Generation via Awareness of Syntactic Dependencies and Semantics (2022.aacl-short)
Copied to clipboard
| Challenge: | Current neural models for Chinese story generation struggle to generate high-quality long text narratives due to ambiguity in syntactically parsing the Chinese language. |
| Approach: | They propose a framework that enhances the feature capturing mechanism by informing the generation model of dependencies between words and additionally augmenting the semantic representation learning through synonym denoising training. |
| Outcome: | The proposed framework outperforms the state-of-the-art Chinese generation models on all evaluation metrics, showing that it enhances dependency and semantic representation learning. |
LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking (2024.eacl-demo)
Copied to clipboard
Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali
| Challenge: | Recent development and success of Large Language Models necessitate evaluation of their performance across diverse NLP tasks in different languages. |
| Approach: | They propose a framework that can be customized to evaluate LLMs for any NLP task, regardless of language. |
| Outcome: | The LLMeBench framework can be customized to evaluate LLMs for any NLP task, regardless of language. |
CRAFT Shared Tasks 2019 Overview — Integrated Structure, Semantics, and Coreference (D19-57)
Copied to clipboard
William Baumgartner, Michael Bada, Sampo Pyysalo, Manuel R. Ciosici, Negacy Hailu, Harrison Pielke-Lombardo, Michael Regan, Lawrence Hunter
| Challenge: | CRAFT corpus provides a unique foundation for integrating natural language processing (NLP) tasks involving structure, semantics, and coreference. |
| Approach: | They propose to use the CRAFT corpus to evaluate three fundamental language processing tasks over full-text biomedical articles. |
| Outcome: | The CRAFT corpus provides a unique foundation for integrating natural language processing tasks involving structure, semantics, and coreference. |
DialogGen: Multi-modal Interactive Dialogue System with Multi-turn Text-Image Generation (2025.findings-naacl)
Copied to clipboard
Minbin Huang, Yanxin Long, Xinchi Deng, Ruihang Chu, Jiangfeng Xiong, Xiaodan Liang, Hong Cheng, Qinglin Lu, Wei Liu
| Challenge: | Text-to-image (T2I) generation models have advanced in recent years, but effective interaction with these models is challenging for average users due to the need for specialized prompt engineering knowledge and the inability to perform multi-turn image generation. |
| Approach: | They propose to use off-the-shelf MLLMs and T2I models to build a multi-modal interactive dialogue system (MIDS) that can generate correct output modalities and coherence of output images. |
| Outcome: | The proposed pipeline can generate correct output modalities and coherent multi-modal outputs compared with other state-of-the-art models. |
Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking (2026.tacl-1)
Copied to clipboard
| Challenge: | Current methods for automated fact-checking rely on relying on other evaluation metrics and closed knowledge sources. |
| Approach: | They propose a method which combines evidence evaluation with verdict-level proxy scoring. |
| Outcome: | The proposed method outperforms existing methods in accuracy and robustness against human ratings and adversarial tests. |
JointCoder: Exploring Automated ICD Coding on Real-World Chinese EHRs with a Multi-Agent Framework (2026.acl-demo)
Copied to clipboard
| Challenge: | Existing automated ICD coding systems face several fundamental challenges due to the limited availability of publicly available Chinese ICD datasets. |
| Approach: | They propose to use a Chinese ICD coding dataset and a multi-agent framework to reformulate ICD as a joint disease-procedure coding task. |
| Outcome: | The proposed system outperforms state-of-the-art methods on real-world Chinese ICD coding datasets and 1.7B-parameter models. |
UScore: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | supervised evaluation metrics are not available for machine translation, despite their wide dissemination. |
| Approach: | They develop fully unsupervised evaluation metrics that leverage parallel data and evaluation metric induction. |
| Outcome: | The proposed metrics beat supervised competitors on 4 out of 5 evaluation datasets. |
STREAM-ZH: Simplified Topic Retrieval Exploration and Analysis Module for Chinese Language (2026.eacl-short)
Copied to clipboard
| Challenge: | Simplified Topic Retrieval Exploration and Analysis Module for Chinese language is the first topic modeling package to fully support the Chinese language. |
| Approach: | They propose a topic modeling package that fully supports the Chinese language . they use preprocessed textual datasets to assess topic models . |
| Outcome: | The proposed framework outperforms existing topic models using English-translated textual input. |
Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics and benchmarks to attribute large language models to structured knowledge are lacking. |
| Approach: | They propose a task of Knowledge-aware Language Model Attribution that improves upon three core concerns with conventional attributed LMs. |
| Outcome: | The proposed model improves upon core concerns with conventional attributed LMs. |
SumPubMed: Summarization Dataset of PubMed Scientific Articles (2021.acl-srw)
Copied to clipboard
| Challenge: | Existing summarization models that can extract the top few lines of news articles fail to summarize long documents. |
| Approach: | They constructed a scientific summarization dataset from MEDLINE articles from the PubMed archive to address this problem. |
| Outcome: | The proposed model outperforms existing models on news article summarization datasets and shows that it is more efficient to extract the top few lines. |
Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking (2022.acl-short)
Copied to clipboard
| Challenge: | Existing evaluation metrics for dialog state tracking are limited for belief states accumulated as dialog proceeds . relative slot accuracy allows intuitive evaluation by assigning relative scores according to the turn of each dialog . |
| Approach: | They propose to use relative slot accuracy to complement existing evaluation metrics . joint goal accuracy and slot accuracy are used to evaluate accumulated belief states . |
| Outcome: | The proposed metrics focus on "penalizing states that fail to predict," not "reward for well-predicted states" the proposed metrics do not depend on the number of predefined slots, and allow intuitive evaluation . |
Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances (2022.acl-short)
Copied to clipboard
| Challenge: | Dialogue State Tracking (DST) is a task-oriented conversational agent that keeps track of key information exchanged during a conversation. |
| Approach: | They propose a new evaluation metric called Flexible Goal Accuracy to address shortcomings of JGA. |
| Outcome: | The proposed metric improves on existing metrics and improves performance of turn-level and non-cumulative belief state models. |
Enhancing Descriptive Image Captioning with Natural Language Inference (2021.acl-short)
Copied to clipboard
| Challenge: | Existing captioning models tend to generate generic captions, but generating descriptive captions is important. |
| Approach: | They propose a novel approach to encourage captioning models to produce more detailed captions using natural language inference. |
| Outcome: | The proposed method outperforms baseline models on MSCOCO metrics on descriptiveness and descriptiveness. |
Translation Canvas: An Explainable Interface to Pinpoint and Analyze Translation Systems (2024.emnlp-demo)
Copied to clipboard
| Challenge: | Existing tools for evaluation of translation models focus on high-level metrics like BLEU or COMET scores, which are time-consuming and prone to error. |
| Approach: | They propose a toolkit that provides a detailed analysis of translation models and a user-friendly interface. |
| Outcome: | The toolkit shows superior performance over COMET and SacreBLEU packages under enjoybility and understandbility criteria. |
SummerTime: Text Summarization Toolkit for Non-experts (2021.emnlp-demo)
Copied to clipboard
Ansong Ni, Zhangir Azerbayev, Mutethia Mutuma, Troy Feng, Yusen Zhang, Tao Yu, Ahmed Hassan Awadallah, Dragomir Radev
| Challenge: | Recent advances in summarization provide models that can generate high quality summaries . a new toolkit for summarizing text is being developed to make it easier for non-experts to keep track of them. |
| Approach: | They develop a toolkit for text summarization that integrates with libraries designed for NLP researchers. |
| Outcome: | SummerTime is a toolkit for text summarization, including models, datasets, and evaluation metrics. |
Text-to-Text Automatic Story Generation: A Survey (2026.eacl-srw)
Copied to clipboard
| Challenge: | Automated story generation aims to produce coherent, engaging, and contextually consistent narratives with minimal or no human involvement . despite advances in large language models, maintaining narrative coherence, character consistency, storyline diversity, and plot controllability in generating stories is still challenging. |
| Approach: | They propose to develop new evaluation metrics and better data sets to support automatic story generation. |
| Outcome: | The proposed evaluation metrics and better datasets will improve narrative coherence and consistency and explore practical applications of story generation. |
A Table-to-Text Framework with Heterogeneous Multidominance Attention and Self-Evaluated Multi-Pass Deliberation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Table-to-text works have been widely applied in different domains, such as weather forecast and financial report generation. |
| Approach: | They propose a table-to-text approach on top of Self-evaluated multi-pass Generation and Heterogenous Multidominance Attention to explore the hierarchical structure. |
| Outcome: | The proposed method outperforms several SOTA methods quantitatively and qualitatively on three public datasets. |
Keyphrase Generation with Fine-Grained Evaluation-Guided Reinforcement Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing KG evaluation metrics are only aware of the exact correctness of predictions on phrase-level and ignore semantic similarities between similar predictions and targets, which inhibits the model from learning deep linguistic patterns. |
| Approach: | They propose a fine-grained evaluation metric to improve the previous KG framework . the evaluation metrics are only aware of the exact correctness of predictions on phrase-level . |
| Outcome: | The proposed method outperforms the existing frameworks among all evaluation scores. |
PILLOW: Enhancing Efficient Instruction Fine-tuning via Prompt Matching (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) has been used to adapt Large Language Models to a variety of tasks, but it requires substantial computational resources to perform. |
| Approach: | They propose a low-rank adaptive learning approach that leverages LoRA's in-context learning capability through prompt matching via reinforcement learning in resource-constrained environments. |
| Outcome: | The proposed model improves LoRA performance on evaluation metrics and utilises consumer-grade GPU resources. |
Abstractive Meeting Summarization: A Survey (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in deep learning have improved language generation systems, opening the door to improved forms of abstractive summarization. |
| Approach: | They propose to use neural encoder-decoder architectures to generate abstractive meeting summarizations that are particularly well-suited for multi-party conversation. |
| Outcome: | The proposed system could be used in a wide variety of real-world contexts, from business meetings to medical consultations to customer service calls. |
Answer-level Calibration for Free-form Multiple Choice Question Answering (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models exhibit few-shot and zero-shot learning capability on tasks such as commonsense reasoning. |
| Approach: | They propose to model context-independent biases in terms of the probability of a choice without the context and to remove it using an unsupervised estimate of similarity with the full context. |
| Outcome: | The proposed model improves over baselines on commonsense reasoning tasks. |
Connecting the Dots: Event Graph Schema Induction with Path Language Modeling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to automate event extraction focus on uncertainty, re-occurring events and multiple hypotheses. |
| Approach: | They propose a new Event Graph Schema where two event types are connected through multiple paths involving entities that fill important roles in a coherent story. |
| Outcome: | The proposed model is highly effective at inducing salient and coherent schemas. |
Towards Reliable Large Audio Language Model (2025.findings-acl)
Copied to clipboard
Ziyang Ma, Xiquan Li, Yakun Song, Wenxi Chen, Chenpeng Du, Jian Wu, Yuanzhe Chen, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen
| Challenge: | Recent advances in large audio language models (LALMs) have demonstrated impressive results and promising prospects in universal understanding and reasoning across speech, music, and general sound. |
| Approach: | They propose to use training-free and training-based methods to enhance LALM reliability to different extents. |
| Outcome: | The proposed methods improve the reliability of large audio language models to different extents. |
Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus (2023.emnlp-main)
Copied to clipboard
Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, Luoyi Fu
| Challenge: | Existing methods for detecting hallucinations in LLMs rely on external knowledge for reference retrieval or require sampling multiple responses for consistency verification. |
| Approach: | They propose a reference-free, uncertainty-based method for detecting hallucinations in Large Language Models that imitates human focus in factuality checking from three aspects: focus on the most informative keywords; focus on unreliable tokens in historical context; focus of token properties such as token type and token frequency. |
| Outcome: | The proposed method achieves state-of-the-art performance across all evaluation metrics and eliminates the need for additional information. |
Revisiting Evaluation of Question Answering Systems in Low-Resource Indic Languages: Bridging Human and Metric Alignment (2026.acl-short)
Copied to clipboard
| Challenge: | Evaluating Question Answering systems in low-resource Indic languages remains challenging due to the scarcity of annotated data and the lack of reliable evaluation metrics. |
| Approach: | They propose a language-based multi-aspect evaluation framework for question answering systems . the framework integrates semantic similarity, factual completeness, numerical accuracy and contextual relevance . |
| Outcome: | The proposed metric is evaluated across eight Indic-language QA tasks using multiple LLMs . Across all settings, it shows stronger agreement with human evaluation . |
PresentAgent: Multimodal Agent for Presentation Video Generation (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Existing methods for generating static slides or text summaries are limited to producing narrated presentations. |
| Approach: | They propose a multimodal agent that transforms long-form documents into narrated presentations. |
| Outcome: | The present agent produces fully synchronized visual and spoken content that closely mimics human-style presentations. |
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark (2022.tacl-1)
Copied to clipboard
| Challenge: | Knowledge-grounded dialogue systems powered by large language models often generate responses that, while fluent, are not attributable to a relevant source of information. |
| Approach: | They propose to evaluate the validity of 12k dialogue turns generated by neural dialogue systems trained on three knowledge-grounded dialogue corpora and to use them to analyze eight evaluation metrics. |
| Outcome: | The proposed evaluation metrics rely on spurious correlations, do not reliably distinguish attributable abstractive responses from unattributable ones, and perform substantially worse when the knowledge source is longer. |
Protein2Text: Resampling Mechanism to Translate Protein Sequences into Human-Interpretable Text (2025.naacl-industry)
Copied to clipboard
Ala Jararweh, Oladimeji Macaulay, David Arredondo, Yue Hu, Luis E Tafoya, Kushal Virupakshappa, Avinash Sahu
| Challenge: | Proteins play critical roles in biological systems, yet 99.7% of 227 million known protein sequences remain uncharacterized due to the limitations of experimental methods. |
| Approach: | They propose a multimodal large language model that interprets protein sequences and generates informative text to address open-ended questions about protein functions and attributes. |
| Outcome: | The proposed model outperforms existing models in open-ended question-answering tasks. |
Large Language Model Evaluation via Matrix Nuclear-Norm (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are computationally intensive due to their O(n3) time complexity with Singular Value Decomposition (SVD). |
| Approach: | They propose a metric to quantify the data compression proficiency of large language models and a convex approximation of matrix rank to capture both predictive discriminability and diversity. |
| Outcome: | The proposed model achieves speeds 8 to 24 times faster than Matrix Entropy for the CEREBRAS-GPT model as models increase from 111M to 6.7B . |
Inconsistent Tokenizations Cause Language Models to be Perplexed by Japanese Grammar (2025.acl-short)
Copied to clipboard
| Challenge: | Standard benchmarks for language models fail to capture nuanced capabilities such as the ability of language models to recognize and obey rare grammar points. |
| Approach: | They find that Weblab's uniformly bad tokenization is a possible root cause for its good performance . |
| Outcome: | The proposed model consistently assigns higher perplexity to ungrammatical psych predicate sentences than grammaticals. |
Explanation Extraction from Hierarchical Classification Frameworks for Long Legal Documents (2024.findings-naacl)
Copied to clipboard
| Challenge: | Hierarchical classification frameworks are black boxes with no explanation for their predictions. |
| Approach: | They develop an extractive explanation algorithm for hierarchical frameworks for long sequences based on the sensitivity of the trained model to input perturbations. |
| Outcome: | The proposed algorithm achieves a minimum gain of 1 point over the previous benchmark on most of the performance metrics. |
Few-Shot Table-to-Text Generation with Prototype Memory (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Neural table-to-text generation models are data-hungry and require large amounts of training data to learn the mapping between tables and texts. |
| Approach: | They propose a framework for table-to-text generation under the few-shot scenario that uses retrieved prototypes and a prototype selector to bridge the structural gap between tables and texts. |
| Outcome: | The proposed framework significantly improves the model performance on three benchmark datasets with state-of-the-art models. |
Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for Grammatical error correction lack explainability . lack of explainability hinders researchers from analyzing strengths and weaknesses of models . |
| Approach: | They propose to assign sentence-level scores to individual edits to improve GEC performance . they use Shapley values, from cooperative game theory, to compute contribution of each edit . |
| Outcome: | The proposed method shows that the evaluation metrics are consistent across edits and human evaluations. |
Rewarding Semantic Similarity under Optimized Alignments for AMR-to-Text Generation (2022.acl-short)
Copied to clipboard
| Challenge: | Automatic evaluation metrics score natural language generation systems based on how well they lexically align to humanannotated references. |
| Approach: | They propose to replace greedy alignments in BERTScore with optimized ones that replace the n-gram matching BERTAcore metrics with a token embedding to prevent domain mismatch. |
| Outcome: | The proposed metrics outperform cross-entropy and BLEU reward baselines on AMR-to-text generation. |
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)
Copied to clipboard
| Challenge: | a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions. |
| Approach: | They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions. |
| Outcome: | The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics. |
Can ChatGPT Assess Human Personalities? A General Evaluation Framework (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies study the virtual personalities of LLMs but rarely explore the possibility of analyzing human personalities via LLM. |
| Approach: | They propose to use Myers–Briggs Type Indicator (MBTI) tests to generate unbiased prompts and replace the subject in question statements to enable flexible queries and assessments. |
| Outcome: | The proposed framework enables LLMs to flexibly assess personalities of different groups of people. |
Beyond Accuracy: Alignment and Error Detection across Languages in the Bi-GSM8K Math-Teaching Benchmark (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in LLMs have significantly improved mathematical problem-solving, with models like GPT-4 achieving human-level performance. |
| Approach: | They propose a bilingual English-Korean dataset enriched with teacher solutions, student solutions, and annotations marking students’ initial errors. |
| Outcome: | The proposed model achieves high agreement with human judgments and lower latency and resource usage than commercial APIs, demonstrating strong computational efficiency. |
ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing question answering datasets for common sense reasoning are lacking for prototypical situations. |
| Approach: | They propose a question answering dataset for training and evaluating common sense reasoning capabilities of artificial intelligence systems in such prototypical situations. |
| Outcome: | The proposed model outperforms existing models on all evaluation metrics with a meaningful gap. |
Unsupervised Disambiguation of Syncretism in Inflected Lexicons (N18-2)
Copied to clipboard
| Challenge: | Lexical ambiguity makes it difficult to compute useful statistics of a corpus. |
| Approach: | They propose a neural network-based model that fits a prior distribution over feature bundles to a list of unigram type counts and partitions each count among different analyses of that unigrammer. |
| Outcome: | The proposed model is based on a list of unigram type counts and partitions each count among different analyses of that unigrammer. |
Revisit Automatic Error Detection for Wrong and Missing Translation – A Supervised Approach (D19-1)
Copied to clipboard
| Challenge: | Current machine translation techniques are bottlenecked by adequacy issues . we propose automatic detection of missing and wrong translations . |
| Approach: | They propose automatic detection of adequacy errors in MT hypothesis for MT model evaluation by annotating missing and wrong translations in 15000 Chinese-English translation pairs. |
| Outcome: | The proposed model can detect missing and wrong translations in 15000 Chinese-English translation pairs. |
QA Analysis in Medical and Legal Domains: A Survey of Data Augmentation in Low-Resource Settings (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized natural language processing, but their success remains limited to high-resource domains. |
| Approach: | They analyze the coverage and representativeness of specialized-domain QA datasets against large-scale reference datasets. |
| Outcome: | The proposed methods and evaluations highlight the challenges faced by LLMs in low-resource domains. |
Analyzing the Performance of Large Language Models on Code Summarization (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models perform very well on tasks that involve both natural language and source code. |
| Approach: | They show that large language models perform very well on tasks that involve both natural language and source code. |
| Outcome: | The proposed models perform very well on tasks that involve both natural language and source code. |
Few-shot learning through contextual data augmentation (2021.eacl-main)
Copied to clipboard
| Challenge: | Various strategies have been explored to learn from a journalist's post-edits . state-of-the-art APE systems require large numbers of post- edits for training . |
| Approach: | They propose to teach a pre-trained machine translation model to translate previously unseen words accurately . they extend a data augmentation approach to create training examples with similar contexts . |
| Outcome: | The proposed model improves accuracy on the scale of one to five examples with only 1 to 5 examples. |
Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inference (2026.findings-eacl)
Copied to clipboard
| Challenge: | Prior studies have examined the impact of structured output on LLMs’ generation quality, often presenting one-way findings. |
| Approach: | They propose to derive five potential causal structures characterizing the influence of structured output on LLMs’ generation using one assumed and two guaranteed constraints. |
| Outcome: | The proposed pipeline can be extended to other modules and is not limited to structured output but can be used in industrial applications. |
Measuring What Counts: The Case of Rumour Stance Classification (2020.aacl-main)
Copied to clipboard
| Challenge: | Numerous methods have been proposed to predict the stance of replies towards a given rumour, but their performance is not optimal for the four-class imbalanced task of rumor stance classification. |
| Approach: | They propose to use a four-class problem to predict the stance of replies towards a given rumour to help identify the most informative minority classes. |
| Outcome: | The proposed methods are robust to imbalanced data and score higher systems capable of recognising the two most informative minority classes (support and deny). |
FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics are not reliable, but require significant computational resources. |
| Approach: | They propose a method to learn a fixed, low cost version of any expensive NLG metric while retaining most of its original performance. |
| Outcome: | The proposed approach retains most of the original performance while running faster and faster. |
Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis often fail due to equipment failure, data corruption, privacy issues and the like. |
| Approach: | They propose a multimodal Transformer framework using prompt learning to address the issue of missing modalities. |
| Outcome: | The proposed framework outperforms existing methods significantly across evaluation metrics. |
Anchor and Broadcast: An Efficient Concept Alignment Approach for Evaluation of Semantic Graphs (2024.lrec-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a sentencelevel formalism designed for English. |
| Approach: | They present an intuitive tool for evaluating graph-based meaning representations . they use an anchor broadcast alignment algorithm that is not subject to local maxima . |
| Outcome: | The proposed tool is highly correlated with the widely used Smatch score, but computation takes only about 40% the time. |
Multi-task Learning for Paraphrase Generation With Keyword and Part-of-Speech Reconstruction (2022.findings-acl)
Copied to clipboard
| Challenge: | PGKPR is a deep learning approach to generate paraphrases using key semantics of the source sentence. |
| Approach: | They propose a model with keyword and part-of-speech reconstruction for paraphrase generation using deep learning. |
| Outcome: | The proposed model outperforms comparative models on two commonly-used datasets. |
Low-resource Entity Set Expansion: A Comprehensive Study on User-generated Text (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing benchmarks for entity set expansion (ESE) are limited to well-formed text and well-defined concepts. |
| Approach: | They propose to use user-generated text to assess the generalizability of ESE methods by identifying phenomena such as non-named entities, multifaceted entities and vague concepts. |
| Outcome: | The proposed methods are based on user-generated text to assess their generalizability and performance. |
From Polarity to Intensity: Mining Morality from Semantic Space (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches to compute moral intensity are limited to word-level measurement and heavily rely on human labelling. |
| Approach: | They propose a weakly-supervised framework that can automatically measure moral intensity from text. |
| Outcome: | The proposed framework can measure moral intensity from text with moral polarity labels, which are more robust and easier to acquire. |
Controlling Text Edition by Changing Answers of Specific Questions (2021.findings-acl)
Copied to clipboard
| Challenge: | In many situations, we need to change specific content in a document. |
| Approach: | They propose a task where we take a long text, a question, and a target answer as input and output a minimally modified text so that it fits the target answer. |
| Outcome: | The proposed task is based on the existing dataset WIKIBIO and tests on a test set. |
Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have not investigated the differences between different correlation measures in meta-evaluation. |
| Approach: | They analyze 12 common correlation measures using real-world data from six widely-used NLG evaluation datasets and 32 evaluation metrics. |
| Outcome: | The proposed measures exhibit the best performance in discriminative power and ranking consistency . the measures using system-level grouping or Kendall correlation are the least sensitive to score granularity . |
What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge (2026.eacl-long)
Copied to clipboard
Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou, Yuan He, Jiaoyan Chen, Steffen Staab, Evgeny Kharlamov
| Challenge: | Existing evaluation metrics and lenient answer matching criteria obscure meaningful comparisons. |
| Approach: | They propose a general method for constructing benchmarks and a method to assess KG-RAG methods under incomplete knowledge. |
| Outcome: | The proposed method systematically assesses KG-RAG methods under incomplete knowledge. |
Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | Abstractive summarization is less prone to unfaithfulness issues than abstractive summaries . but, unfaitfulness problems, i.e., hallucinating new information, are still a problem in extractive summarisation . |
| Approach: | They propose a typology with five types of broad unfaithfulness problems that can appear in extractive summaries, including and beyond not-entailment. |
| Outcome: | The proposed metric shows that it detects unfaithful summaries faster than existing faithfulness evaluation metrics. |
Examining the State-of-the-Art in News Timeline Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on news timeline summarization (TLS) has left an unclear picture of how well it is currently solved and how it can be approached. |
| Approach: | They propose a combination of different TLS strategies that improves over the stateof-the-art on all tested benchmarks. |
| Outcome: | The proposed method improves over the state-of-the-art on all tested benchmarks. |
Exemplar Encoder-Decoder for Neural Conversation Generation (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to generate conversational systems suffer from lack of diversity in responses and generation of short, repetitive and uninteresting responses. |
| Approach: | They propose a novel conversation model that uses similar examples from training data to generate responses. |
| Outcome: | The proposed model outperforms state-of-the-art sequence to sequence learning on several evaluation metrics on two large data sets. |
LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation are expensive and time-consuming. |
| Approach: | They propose a framework that utilizes LLMs to generate synthetic evaluation datasets . they propose meta-correlation to measure alignment between metric rankings and human benchmarks based on synthetic data . |
| Outcome: | The proposed framework achieves meta-correlations exceeding 0.9 in multilingual QA and replaces human judgment with synthetic evaluation datasets. |
Are Your Keywords Like My Queries? A Corpus-Wide Evaluation of Keyword Extractors with Real Searches (2025.coling-main)
Copied to clipboard
| Challenge: | Keyword Extraction (KE) is essential in Natural Language Processing (NLP) for identifying key terms that represent the main themes of a text. |
| Approach: | They propose to use real query data from Google Trends to evaluate keywords extracted from a text to capture users' top queries. |
| Outcome: | The proposed method can be used with both supervised and unsupervised KE approaches and shows that KeyBERT is the most effective in capturing users’ top queries. |
Unsupervised Text Style Transfer for Controllable Intensity (2026.findings-eacl)
Copied to clipboard
| Challenge: | Unsupervised Text Style Transfer (UTST) aims to transfer the stylistic properties of a given text without parallel text pairs. |
| Approach: | They propose a SFT-then-PPO paradigm to fine-tune an LLM with parallel data and reward functions for distinguishing stylistic intensity in hierarchical levels. |
| Outcome: | The proposed system can transfer stylistic properties without parallel text pairs even for adjacent levels of intensity. |
Promoting Topic Coherence and Inter-Document Consorts in Multi-Document Summarization via Simplicial Complex and Sheaf Graph (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems that generate summaries from multiple sources often lack accuracy and accuracy due to the length of tokens used in encoding. |
| Approach: | They propose a novel encoder-decoder model that uses pre-trained BART to analyze linguistic nuances, simplicial complex layer to apprehend inherent properties that transcend pairwise associations and sheaf graph attention to effectively capture heterophilic properties. |
| Outcome: | The proposed model achieves consistent performance improvement across all evaluation metrics (syntactical, semantical and faithfulness). |
Rethinking Sentiment Style Transfer (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods for text style transfer are unsatisfactory. |
| Approach: | They propose to use a graph-based method to extract attribute content from sentences . they propose an efficient regularization to leverage attribute-dependent content as guiding signals. |
| Outcome: | The proposed method is based on a YELP and IMDB dataset and it is able to detect errors in the human evaluation. |
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization (2021.acl-short)
Copied to clipboard
| Challenge: | Experimental results show that SimCLS can improve existing top-performing models by a large margin. |
| Approach: | They propose a framework for abstractive summarization that is conceptually simple and empirically powerful. |
| Outcome: | The proposed framework improves the performance of top-performing models by a large margin against existing top-scoring systems. |
In-Context Former: Lightning-fast Compressing Context for Large Language Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to reduce inference costs of transformer-based large language models entail quadratic complexity . et al., 2017): transformer-derived large language model performance is a major challenge. |
| Approach: | They propose a method that compresses long contexts into short soft prompts . they use the self-attention mechanism of the large model to extract and condense information . |
| Outcome: | The proposed method reduces compression costs by 68 to 112 times while achieving 90% of baseline performance. |
IMO: Greedy Layer-Wise Sparse Representation Learning for Out-of-Distribution Text Classification with Pre-trained Models (2024.acl-long)
Copied to clipboard
| Challenge: | IMO is a machine learning model that learns invariant features from unseen domains. |
| Approach: | They propose IMO: Invariant features Masks for Out-of-Distribution text classification to achieve OOD generalization by learning invariant feature masks. |
| Outcome: | The proposed model outperforms baseline models in various evaluation metrics and settings. |
Long Text and Multi-Table Summarization: Dataset and Method (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing document summarization methods focus on the text and filter out the non-textual content. Existing methods cannot meet the requirements of summarizing long text and multiple tables in each report. |
| Approach: | They propose a dataset for automatic document summarization that uses text and tabular data to produce a concise summary covering the input document's salient information. |
| Outcome: | The proposed method can produce a concise summary covering the input document's salient information. |
ICE-Score: Instructing Large Language Models to Evaluate Code (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in the field of natural language generation have facilitated the use of large language models to assess the quality of generated text. |
| Approach: | They propose a new evaluation metric by instructing large language models for code assessments using a set of programming languages. |
| Outcome: | The proposed evaluation metric surpasses state-of-the-art metrics for code generation, delivering high levels of accuracy and consistency across programming languages and tasks. |
A Survey on Asking Clarification Questions Datasets in Conversational Systems (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies on Asking Clarification Questions (ACQs) are incomparable due to inconsistent data, experimental setups and evaluation strategies. |
| Approach: | They analyse the current research status on Asking Clarification Questions (ACQs) and propose a set of evaluation metrics and benchmarks for multiple ACQs-related tasks. |
| Outcome: | The proposed techniques are compared with the available datasets and evaluated against benchmarks. |
On Context Utilization in Summarization with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models excel in abstractive summarization tasks, delivering fluent and pertinent summaries. |
| Approach: | They conduct the first comprehensive study on context utilization and position bias in summarization. |
| Outcome: | The proposed benchmark compares two methods to alleviate position bias in summarization tasks. |
Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Time (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing LLMs struggle to reliably detect subtle reasoning errors in ASAS tasks. |
| Approach: | They propose a dual-model framework with a dedicated Critic model trained for effective reflection that generates precise verbal feedback. |
| Outcome: | The proposed framework outperforms existing ASAS benchmarks and provides valuable insights into the performance of the proposed framework. |
End-to-End Construction of NLP Knowledge Graph (2021.findings-acl)
Copied to clipboard
| Challenge: | a new schema for NLP knowledge about tasks, datasets and metrics is proposed. |
| Approach: | They propose a new schema that represents knowledge about tasks, datasets and metrics in the NLP domain. |
| Outcome: | The proposed framework can be automatically built into scientific leaderboards . the proposed system achieves reasonable results for all relation types on this small-scale graph . |
An Empirical Study of Clinical Note Generation from Doctor-Patient Encounters (2023.eacl-main)
Copied to clipboard
| Challenge: | Medical doctors spend 52 to 102 minutes per day writing clinical notes from patient encounters. |
| Approach: | They propose to use a new dataset to generate automated and manual clinical notes from doctor-patient conversations in a clinical setting. |
| Outcome: | The proposed model could reduce the time spent writing clinical notes from doctor-patient conversations in a clinical setting. |
An Evaluation of Disentangled Representation Learning for Texts (2021.findings-acl)
Copied to clipboard
| Challenge: | Disentangled representations of texts encode information pertaining to different aspects of the text in separate vector embeddings. |
| Approach: | They propose to use a highly-structured natural language dataset to evaluate disentangled representations for texts. |
| Outcome: | The proposed models are well-suited for learning disentangled representations of texts on a synthetic natural language dataset. |
A Dog Is Passing Over The Jet? A Text-Generation Dataset for Korean Commonsense Reasoning and Evaluation (2022.findings-naacl)
Copied to clipboard
Jaehyung Seo, Seounghoon Lee, Chanjun Park, Yoonna Jang, Hyeonseok Moon, Sugyeong Eo, Seonmin Koo, Heuiseok Lim
| Challenge: | Korean pretrained language models struggle to generate short sentences with a given condition based on compositionality and commonsense reasoning. |
| Approach: | They propose a Korean text-generation dataset for Korean generative commonsense reasoning and language model evaluation using a semi-automatic dataset construction approach. |
| Outcome: | The proposed dataset is available at http://aihub.or.kr/opendata/korea-university. |
Is the Best Better? Bayesian Statistical Model Comparison for Natural Language Processing (2020.emnlp-main)
Copied to clipboard
| Challenge: | a recent study raises concerns about the use of standard splits to compare models . we compare the performance of six English part-of-speech taggers to those of other models based on standard split analysis . |
| Approach: | They propose a Bayesian statistical model comparison technique using k-fold cross-validation . they rank six English part-of-speech taggers across two data sets and three evaluation metrics . |
| Outcome: | The proposed method ranks English part-of-speech taggers on two data sets and three evaluation metrics. |
Better than Average: Paired Evaluation of NLP systems (2021.acl-long)
Copied to clipboard
| Challenge: | Evaluation in NLP is usually done by comparing the scores of competing systems . averaging scores independently and declaring the best system is difficult . |
| Approach: | They examine the use of averages to aggregate evaluation scores into a final number . they argue that the average ignores the pairing arising from the fact that systems are evaluated on the same test instances. |
| Outcome: | The proposed method ignores the pairing arising from the fact that systems are evaluated on the same test instances. |
DataSciBench: An LLM Agent Benchmark for Data Science (2026.findings-acl)
Copied to clipboard
Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, Yisong Yue
| Challenge: | Existing benchmarks focus on single task, simple evaluation metrics, and readily available ground truth (GT) DataSciBench is built on curated, natural, and challenging prompts with complex evaluation criteria and uncertain GT. |
| Approach: | They propose a benchmark for evaluating Large Language Models in data science that integrates LLM-based self-consistency and human verification to ensure accuracy. |
| Outcome: | The proposed framework outperforms open-source models in all metrics and offers rigorous insights into LLM strengths and weaknesses. |
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation (P19-1)
Copied to clipboard
| Challenge: | Existing metrics for vision-and-language navigation focus on goal completion rather than the sequence of actions corresponding to the instructions. |
| Approach: | They propose to use a room-to-room dataset to measure the length of instruction followed by agents. |
| Outcome: | The proposed metric outperforms existing metrics for Room-to-Room tasks because it is direct-to goal shortest. |
Looking at Radiology Report Generation through a Causal Lens: A Survey (2026.acl-long)
Copied to clipboard
| Challenge: | Existing surveys on RRG emphasize deep learning while overlooking the critical role of causality. |
| Approach: | They propose to analyze biases across the RRG pipeline and formalize it as a causal modeling problem and review representative causal techniques from the literature. |
| Outcome: | The proposed model can mitigate biases and yield fair, reliable systems with clinically meaningful outputs. |
A Survey on Zero Pronoun Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Zero pronouns (ZPs) are often omitted in pro-drop languages, but should be recalled in non-pro-drop language. |
| Approach: | They propose to analyze the literature on zero pronoun translation after the neural revolution . they uncover that data limitation causes learning bias in languages and domains . |
| Outcome: | The proposed method and methods are compared to other models and evaluation metrics on different benchmarks. |
Video Question Answering with Phrases via Semantic Roles (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing VidQA evaluation metrics limit the models’ application scenario to a single-word answer or selecting a phrase from a fixed set of phrases. |
| Approach: | They propose to leverage video descriptions to mask out certain phrases to enable evaluation of answer phrases. |
| Outcome: | The proposed model reduces the influence of language bias on VidQA datasets by retrieving a video having a different answer for the same question. |
Learning to Select, Track, and Generate for Data-to-Text (P19-1)
Copied to clipboard
Hayate Iso, Yui Uehara, Tatsuya Ishigaki, Hiroshi Noji, Eiji Aramaki, Ichiro Kobayashi, Yusuke Miyao, Naoaki Okazaki, Hiroya Takamura
| Challenge: | Existing models often refer to the same data record multiple times. |
| Approach: | They propose a data-to-text generation model with two modules, one for tracking and the other for text generation. |
| Outcome: | The proposed model outperforms existing models even without writer information in all evaluation metrics and contributes to content planning and surface realization. |
Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve self-correction performance of Large Language Models are based on intrinsic selfcorrectione, which allows the model to check and revise its selfgenerated answers without external feedback. |
| Approach: | They propose to decompose the self-correction capability into confidence and critique capabilities and a metric for overall self-corretion capability evaluation. |
| Outcome: | The proposed method outperforms vanilla SFT and achieves much higher accuracy after self-correction. |
CodeTaxo: Enhancing Taxonomy Expansion with Limited Examples via Code Language Prompts (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing taxonomies are mainly constructed by experts or through crowd-sourcing, making the process time-consuming, labor-intensive, and restricted in coverage. |
| Approach: | They propose a method that leverages large language models to capture taxonomic structure . existing taxonomies are mainly constructed by experts or through crowd-sourcing . |
| Outcome: | Experiments on five real-world domains show that CodeTaxo outperforms state-of-the-art methods. |
Linguistic Appropriateness and Pedagogic Usefulness of Reading Comprehension Questions (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing evaluation measures for automatic generation of reading comprehension questions focus on linguistic quality only, ignoring educational value and appropriateness of questions. |
| Approach: | They propose a new evaluation scheme where questions are structured in a hierarchical way . they also create and evaluate two new evaluation data sets for Basque and German . |
| Outcome: | The proposed evaluation scheme can be applied, but expert annotators are needed. |
AudioBench: A Universal Benchmark for Audio Large Language Models (2025.naacl-long)
Copied to clipboard
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, Nancy F. Chen
| Challenge: | Existing evaluation regimes for audio large language models do not cover the breadth of their possible use cases. |
| Approach: | They propose to use AudioBench to evaluate audio large language models . they found that no single model excels consistently across all tasks . |
| Outcome: | The proposed evaluation targets speech understanding, audio scene understanding, and voice understanding (paralinguistic) . no single model excels consistently across all tasks, the paper found . |
A Top-down Graph-based Tool for Modeling Classical Semantic Maps: A Case Study of Supplementary Adverbs (2025.naacl-long)
Copied to clipboard
| Challenge: | Semantic map models (SMMs) construct a network-like conceptual space from cross-linguistic instances or forms based on the connectivity hypothesis. |
| Approach: | They propose a graph-based algorithm that automatically generates conceptual spaces and SMMs in a top-down manner. |
| Outcome: | The proposed model is compared with human annotations and other automated methods on cross-linguistic supplementary adverbs. |
Latent Code and Text-based Generative Adversarial Networks for Soft-text Generation (N19-1)
Copied to clipboard
| Challenge: | Text generation with generative adversarial networks (GANs) can be divided into text-based and code-based categories depending on the type of signals used for discrimination. |
| Approach: | They propose a text-based approach to exploit generative adversarial networks (GANs) by using autoencoders to provide a continuous representation of sentences, which they will refer to as soft-text, and hybrid latent code and text-oriented approaches with one or more discriminators. |
| Outcome: | The proposed approach outperforms the traditional GAN-based methods on two well-known datasets. |
Stylized Knowledge-Grounded Dialogue Generation via Disentangled Template Rewriting (2022.naacl-main)
Copied to clipboard
Qingfeng Sun, Can Xu, Huang Hu, Yujing Wang, Jian Miao, Xiubo Geng, Yining Chen, Fei Xu, Daxin Jiang
| Challenge: | Existing knowledge-grounded dialogue generation models only produce pedantic responses, which lacks emotion and attraction compared with the responses with polite style, positive and negative sentiments. |
| Approach: | They propose a method which generates responses via combing disentangled style templates and content templates. |
| Outcome: | The proposed method improves on evaluation metrics compared with state-of-the-art methods. |
SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Empirical studies show that supervised learning is extremely effective in in-domain datasets and models trained on SuperDialseg can achieve good generalization ability on out-of-domain data. |
| Approach: | They propose a supervised definition of dialogue segmentation points using document-grounded dialogues and a large-scale supervised dataset called SuperDialseg. |
| Outcome: | The proposed model can achieve good generalization ability on out-of-domain data. |
Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents (2025.emnlp-main)
Copied to clipboard
Haochen Sun, Shuwen Zhang, Lujie Niu, Lei Ren, Hao Xu, Hao Fu, Fangkun Zhao, Caixia Yuan, Xiaojie Wang
| Challenge: | Large Language Models (LLMs) based agent systems have made great strides in real-world applications beyond traditional NLP tasks. |
| Approach: | They propose a new LLM-based Multi-Agent System benchmark, Collab-Overcooked, built on the popular Overcooked-AI game with more applicable and challenging tasks in interactive environments. |
| Outcome: | The proposed benchmark provides a multi-agent framework supporting diverse tasks and objectives and encourages collaboration through natural language communication. |
Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models (2025.coling-main)
Copied to clipboard
Anmol Reddy Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid A. Hasan, Elita A.A Lobo
| Challenge: | Existing methods for large language models rely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. |
| Approach: | They propose an approach which combines negative feedback with in-domain positive feedback on the forget set and introduces new evaluation metrics to assess the quality of responses related to the forget sets. |
| Outcome: | The proposed approach avoids undesirable model behaviors while maintaining overall model performance. |
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (D19-1)
Copied to clipboard
| Challenge: | Existing QG models suffer from a “semantic drift” problem, i.e., the semantics of the model-generated question drifts away from the given context and answer. |
| Approach: | They propose two semantics-enhanced rewards obtained from downstream question paraphrasing and question answering tasks to regularize the QG model to generate semantically valid questions. |
| Outcome: | The proposed method achieves state-of-the-art performance w.r.t. traditional evaluation metrics and performs best on QA-based evaluation metrics. |
CASIMIR: A Corpus of Scientific Articles Enhanced with Multiple Author-Integrated Revisions (2024.lrec-main)
Copied to clipboard
| Challenge: | CASIMIR dataset contains multiple revisions of 15,646 scientific articles . authors question the relevance of current evaluation methods for text revision . |
| Approach: | They propose a textual resource on the revision step of the writing process of scientific articles. |
| Outcome: | The proposed dataset contains the multiple revised versions of 15,646 scientific articles from OpenReview, along with their peer reviews. |
DP-Rewrite: Towards Reproducibility and Transparency in Differentially Private Text Rewriting (2022.coling-1)
Copied to clipboard
| Challenge: | Existing systems for differentially private text rewriting lack the means to validate privacy-preserving claims. |
| Approach: | They propose an open-source framework for differentially private text rewriting which is modular, extensible and highly customizable. |
| Outcome: | The proposed framework provides a way to lead and validate private text rewriting research. |
Mitigating Selection Bias with Node Pruning and Auxiliary Options (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models exhibit systematic preferences for answer choices when answering multiple-choice questions. |
| Approach: | They propose two methods to identify and remove internal sources of selection bias . they propose Choice Kullback-Leibler Divergence (CKLD) to capture distributional imbalances in model predictions. |
| Outcome: | The proposed methods improve answer accuracy while reducing selection bias. |
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2022.naacl-main)
Copied to clipboard
Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander Fabbri, Yejin Choi, Noah A. Smith
| Challenge: | Recent advances on models and metrics should benefit and inform each other, authors argue . bidimensional leaderboards allow for fast, accurate evaluation of language generation models . |
| Approach: | They propose a bidimensional leaderboard that tracks progress in language generation models and metrics for their evaluation. |
| Outcome: | The proposed leaderboards track progress in language generation models and metrics for their evaluation. |
Automated Progressive Red Teaming (2025.coling-main)
Copied to clipboard
| Challenge: | Automated red teaming (ART) is effective but time-consuming, costly and lacks scalability. |
| Approach: | They propose an automated red teaming framework that generates adversarial prompts to expose LLM vulnerabilities. |
| Outcome: | The proposed framework explores and exploits LLM vulnerabilities through multi-round interactions. |
Revisiting non-English Text Simplification: A Unified Multilingual Benchmark (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in English automatic text simplification have pushed the frontier of multilingual text simulating. |
| Approach: | They propose to use multilingual evaluation benchmarks to evaluate multilingual text simplification models in English and other languages. |
| Outcome: | The proposed benchmark outperforms pre-trained models in Russian in zero-shot cross-lingual transfer to low-resource languages. |
Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to metric meta-evaluation focus on general statements about absolute and relative quality of metrics across arbitrary system outputs, but in practice, metrics are applied in highly contextual settings. |
| Approach: | They propose a method for contextual metric meta-evaluation by comparing local metric accuracy. |
| Outcome: | The proposed method compares the local metric accuracy of evaluation metrics across translation, speech recognition, and ranking tasks. |
DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence (2023.eacl-main)
Copied to clipboard
| Challenge: | DiscoScore is a parametrized discourse metric that uses BERT to model discourse coherence . it is weak when operated at system level, and is therefore not reliable in a way to spot improvements . |
| Approach: | They propose a parametrized discourse metric which uses BERT to model discourse coherence from different perspectives. |
| Outcome: | The proposed model outperforms existing models on document-level machine translation and summarization. |
Progress Ratio Embeddings: An Impatience Signal for Robust Length Control in Neural Text Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Modern neural language models achieve high accuracy in text generation, yet precise control over generation length remains underdeveloped. |
| Approach: | They propose a method to provide robust length control using Reverse Positional Embeddings. |
| Outcome: | The proposed method provides stable length fidelity without degrading text accuracy . the proposed method generalizes well to unseen target lengths . |
Modeling Coreference Relations in Visual Dialog (2021.eacl-main)
Copied to clipboard
| Challenge: | Visual dialog is a vision-language task where an agent needs to answer a series of questions grounded in an image based on the understanding of the dialog history and the image. |
| Approach: | They propose two soft constraints that can improve the model’s ability of resolving coreferences in dialog in an unsupervised way based on linguistic knowledge and discourse features of human dialog. |
| Outcome: | The proposed model achieves state-of-the-art performance on the VisDial v1.0 dataset without pretraining on other vision language datasets. |
KEBAP: Korean Error Explainable Benchmark Dataset for ASR and Post-processing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conventional evaluation metrics for automatic speech recognition systems produce a singular aggregate score, which is insufficient for understanding specific system vulnerabilities. |
| Approach: | They propose to introduce the Korean Error Explainable Benchmark Dataset for ASR and Post-processing (KEBAP) this method enables a more balanced assessment encompassing speech recognition accuracy and user readability. |
| Outcome: | The proposed method enables a more balanced assessment encompassing speech recognition accuracy and user readability. |
Where is this coming from? Making groundedness count in the evaluation of Document VQA models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Document Visual Question Answering (VQA) models have come close to or matching human performance on some benchmarks. |
| Approach: | They propose a method that accounts for the semantic and multimodal groundedness of a model’s outputs and can be parameterized so that users can configure the score according to their preferences. |
| Outcome: | The proposed method produces scores that are a better indicator of a model’s robustness and tends to give higher rewards to better-calibrated answers. |
ReLearn: Unlearning via Learning for Large Language Models (2025.acl-long)
Copied to clipboard
Haoming Xu, Ningyuan Zhao, Liming Yang, Sendong Zhao, Shumin Deng, Mengru Wang, Bryan Hooi, Nay Oo, Huajun Chen, Ningyu Zhang
| Challenge: | Existing methods for unlearning large language models often rely on reverse optimization to reduce target token probabilities. |
| Approach: | They propose a data augmentation and fine-tuning pipeline for effective unlearning . they propose augmentation, evaluation frameworks to measure contextual forgetting . |
| Outcome: | The proposed framework achieves targeted forgetting while preserving high-quality outputs. |
An Empirical Comparison of Unsupervised Constituency Parsing Methods (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for unsupervised constituency parsing are inconsistent due to data preprocessing, lexicalization, and evaluation metrics. |
| Approach: | They propose to standardize experimental settings for better comparability between methods . they compare existing methods with those proposed by decade-old models . |
| Outcome: | The proposed methods perform better than decade-old models on English and Japanese, respectively, compared with decade- old models. |
Interpretable Research Replication Prediction via Variational Contextual Consistency Sentence Masking (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for predicting research replication are insufficient especially for long research papers. |
| Approach: | They propose to build an interpretable neural model which can provide sentence-level explanations and apply weakly supervised approach to leverage large corpus of unlabeled datasets. |
| Outcome: | The proposed model can provide sentence-level explanations and leverage large unlabeled datasets to boost interpretability and improve prediction performance. |
SDA: Semantic Discrepancy Alignment for Text-conditioned Image Retrieval (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for textconditioned image retrieval are limited due to the scale of training and the complexity of attributes in modification texts. |
| Approach: | They propose a general boosting framework to address these issues by employing semantic discrepancy alignment by formulating distance consistency and neighbor consistency between the image and text domains. |
| Outcome: | The proposed framework improves retrieval performance on three prominent datasets with state-of-the-art results. |
Trigger-Argument based Explanation for Event Detection (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing works on ED use words or phrases to explain models’ inner mechanisms, but for ED, the event structure is more enlightening clues to explain model behaviors. |
| Approach: | They propose a Trigger-Argument based Explanation method which can utilize event structure knowledge to uncover a faithful interpretation for existing ED models at neuron level. |
| Outcome: | The proposed method can reveal the process by which the model predicts on the large-scale MAVEN and the widely-used ACE 2005 datasets. |
MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents (2026.acl-long)
Copied to clipboard
Pengxiang Zhao, Guangyi Liu, Yaozhen Liang, Weiqing He, Zhengxi Lu, WenHao Wang, Yuehao Huang, Yuxiang Chai, Zhaolu Kang, Yaxuan Guo, Hao Wang, Kexin Zhang, Liang Liu, Yong Liu
| Challenge: | Shortcuts such as APIs and deep-links have emerged as efficient complements to flexible GUI operations, but systematic evaluation of GUI–shortcut hybrid agents remains underexplored. |
| Approach: | They propose a benchmark that evaluates GUI-shortcut hybrid agents with a specific focus on the mobile domain. |
| Outcome: | MAS-Bench evaluates agent's ability to generate shortcuts by discovering and creating reusable, low-cost workflows. |
Counterspeeches up my sleeve! Intent Distribution Learning and Persistent Fusion for Intent-Conditioned Counterspeech Generation (2023.acl-long)
Copied to clipboard
| Challenge: | a counterspeech with a certain intent may not be sufficient in every situation due to complex nature of hate speech . a novel framework for intent-conditioned counterseech generation is proposed to address the pervasive issue of hateful speech on the internet. |
| Approach: | They propose a framework for intent-conditioned counterspeech generation that leverages intent-specific representations and a fusion module to incorporate intent-related information into the model. |
| Outcome: | The proposed framework outperforms baselines by 10% across evaluation metrics. |
Towards Explainable Evaluation of Language Models on the Semantic Similarity of Visual Concepts (2022.coling-1)
Copied to clipboard
Maria Lymperaiou, George Manoliadis, Orfeas Menis Mastromichalakis, Edmund G. Dervakos, Giorgos Stamou
| Challenge: | Recent advances in NLP research have focused on robustness and explainability issues of their evaluation strategies. |
| Approach: | They propose to use pre-trained transformers to evaluate semantic similarity for visual vocabularies . they propose to provide explainable metrics for understanding the quality of retrieved instances . |
| Outcome: | The proposed metrics highlight inabilities of widely used evaluation methods and highlight weaknesses in learned linguistic representations. |
QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics poorly approximate parser quality, says a new study . questions under discussion is a linguistic framework that views discourse as asking questions and answering them . |
| Approach: | They propose a framework for automatic evaluation of QUD parsing . they use a dataset of fine-grained evaluation of 2,190 QUD questions . |
| Outcome: | The proposed framework shows that satisfying constraints of QUD is still challenging for modern LLMs. |
Measuring the Robustness of Reference-Free Dialogue Evaluation Systems (2025.coling-main)
Copied to clipboard
| Challenge: | Advancements in dialogue systems powered by large language models have outpaced the development of reliable evaluation metrics. |
| Approach: | They propose a benchmark to evaluate the robustness of reference-free dialogue metrics against four categories of adversarial attacks. |
| Outcome: | The proposed benchmarks show that the two axes of reliability are not always aligned . the findings motivate the development of nuanced evaluation frameworks to address real-world dialogue challenges. |
Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations (2022.acl-long)
Copied to clipboard
Wei Chen, Yeyun Gong, Can Xu, Huang Hu, Bolun Yao, Zhongyu Wei, Zhihao Fan, Xiaowu Hu, Bartuer Zhou, Biao Cheng, Daxin Jiang, Nan Duan
| Challenge: | Existing studies focus on coarse-grained response selection in retrieval-based dialogue systems. |
| Approach: | They propose a Contextual Fine-to-Coarse (CFC) distilled model for coarse-grained response selection in open-domain conversations. |
| Outcome: | The proposed model improves over baseline methods on two datasets based on the Reddit comments dump and Twitter corpus compared with baseline methods. |
LegalViz: Legal Text Visualization by Text To Diagram Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Graphviz provides diagrams for legal documents that are easy to understand and understand . a novel dataset of 23 languages and 7,010 cases of legal document and visualization pairs is proposed . |
| Approach: | They propose a dataset of legal diagrams using DOT graph description language of Graphviz. |
| Outcome: | The proposed dataset outperforms existing models including GPTs in 23 languages and 7,010 cases of legal document and visualization pairs. |
AGIC: Attention-Guided Image Captioning to Improve Caption Relevance (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for image captioning generate generic captions that are limited in capturing nuanced visual details. |
| Approach: | They propose attention-guided image captioning which amplifies visual regions directly in the feature space to guide caption generation. |
| Outcome: | The proposed approach matches or surpasses state-of-the-art models while achieving faster inference. |
Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to generate implausible stories using plots are unnatural and oversimplify the characteristics of implusible machine-generated stories. |
| Approach: | They propose to generate a more comprehensive set of implausible stories using plots . plots are structured representations of controllable factors used to generate stories . |
| Outcome: | The proposed model improves the quality of generated implausible stories using plots . it shows that the evaluation metrics trained on the generated data correlate better with human judgments compared to baselines. |
A Comparative Study of Faithfulness Metrics for Model Interpretability Methods (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to reveal the reasoning processes of machine learning models are difficult to interpret due to their complexity. |
| Approach: | They propose to use diagnosticity and complexity to assess faithfulness of machine learning models . they propose to apply posthoc interpretation methods to reveal reasoning behind models based on internal reasoning . |
| Outcome: | The proposed interpretation metrics show conflicting preferences when comparing interpretations . sufficiency and comprehensiveness metrics have higher diagnosticity and lower complexity . |
CogGPT: Unleashing the Power of Cognitive Dynamics on Large Language Models (2024.findings-emnlp)
Copied to clipboard
Yaojia Lv, Haojie Pan, Zekun Wang, Jiafeng Liang, Yuanxing Liu, Ruiji Fu, Ming Liu, Zhongyuan Wang, Bing Qin
| Challenge: | Recent advances in large language models (LLMs) focus on replicating human cognition in specific contexts, overlooking the inherently dynamic nature of cognition. |
| Approach: | They propose a task to assess cognitive dynamics of large language models (LLMs) they introduce a benchmark and two evaluation metrics to validate the benchmark and evaluate it through participant surveys. |
| Outcome: | The proposed task overcomes the limitations of existing methods and is available for download. |
DeltaScore: Fine-Grained Story Evaluation with Perturbations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics for stories are limited in assessing intricate aspects of storytelling, such as fluency and interestingness. |
| Approach: | They propose a novel method that uses perturbation techniques to evaluate story aspects . they compare fluency, coherence, relatedness, logicality, interestingness and interestingness to existing metrics . |
| Outcome: | The proposed method shows that one specific perturbation is highly effective in capturing multiple aspects. |
An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results (2020.acl-main)
Copied to clipboard
| Challenge: | Existing Ordinal Classification metrics ignore the ordering between items or assume additional information. |
| Approach: | They propose a Closeness Evaluation Measure for Ordinal Classification based on Measurement Theory and Information Theory. |
| Outcome: | The proposed metric captures quality aspects from different traditional tasks simultaneously. |
A Co-Attentive Cross-Lingual Neural Model for Dialogue Breakdown Detection (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for dialogue breakdown detection do not focus on preventing dialogue breakdowns. |
| Approach: | They propose a model that integrates a pretrained cross-lingual language model and a co-attention network for dialogue breakdown detection. |
| Outcome: | The proposed model outperforms all previous approaches on evaluation metrics in Japanese and English tracks in Dialogue Breakdown Detection Challenge 4 . |
GreedyCAS: Unsupervised Scientific Abstract Segmentation with Normalized Mutual Information (2023.emnlp-main)
Copied to clipboard
| Challenge: | Abstracts of scientific papers typically contain premises and conclusions, but in non-structured abstracts the concluding information is not marked. |
| Approach: | They propose to use Normalized Mutual Information (NMI) to optimize the NMI score between two segments by assuming that conclusions are strongly semantically linked with preceding premises. |
| Outcome: | The proposed approach outperforms baseline methods on structured abstracts and on non-structured abstracts. |
CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating the performance of Grammatical Error Correction systems is a challenging task due to its subjectivity. |
| Approach: | They propose a method to evaluate GEC systems in multi-reference evaluation setting . they use consistent edit boundaries to eliminate bias caused by inconsistent edit boundaries . |
| Outcome: | The proposed evaluation metric eliminates bias caused by inconsistent edit boundaries on six English reference sets. |
Beyond Performance: Quantifying and Mitigating Label Bias in LLMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit undesirable preference toward predicting certain answers over others, despite their adaptability to diverse tasks. |
| Approach: | They propose a label bias calibration method that outperforms recent calibration approaches for improving performance and mitigating label bias. |
| Outcome: | The proposed method outperforms calibration approaches for improving performance and mitigating label bias. |
Generating Textual Explanations for Machine Learning Models Performance: A Table-to-Text Task (2022.lrec-1)
Copied to clipboard
| Challenge: | Numerical tables are widely used to communicate or report the classification performance of machine learning models with respect to a set of evaluation metrics. |
| Approach: | They propose a task where neural models are trained to generate textual explanations based on the metrics’ scores reported in numerical tables. |
| Outcome: | The proposed model outperforms existing methods and can be used to explain the performance of ML models. |
Pulling Out All The Full Stops: Punctuation Sensitivity in Neural Machine Translation and Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Several studies have documented the sensitivity of machine translation models to noise, including spelling errors and dialects. |
| Approach: | They propose to test machine translation systems for punctuation sensitivity by performing sentence-final perturbation tests with full stops, exclamation and questions marks. |
| Outcome: | The proposed tests show that commercial translation systems are vulnerable to single punctuation insertion or deletion, resulting in unreliable translations. |
Evaluation of Question Generation Needs More References (2023.findings-acl)
Copied to clipboard
Shinhyeok Oh, Hyojun Go, Hyeongdon Moon, Yunsung Lee, Myeongho Jeong, Hyun Seung Lee, Seungtaek Choi
| Challenge: | Existing evaluations of QG methods rely on single reference-based similarity metrics . multiple (pseudo) references are more effective for QG evaluation . |
| Approach: | They propose to paraphrase the reference question for a more robust QG evaluation. |
| Outcome: | The proposed frameworks show higher correlation with human evaluations than evaluation with a single reference. |
Evaluating Extreme Hierarchical Multi-label Classification (2022.acl-long)
Copied to clipboard
| Challenge: | Several natural language processing tasks are defined as a classification problem in its most complex form: Multi-label Hierarchical Extreme classification. |
| Approach: | They propose a classification metric inspired by the Information Contrast Model (ICM) they use a set of formal properties to analyze the evaluation metrics. |
| Outcome: | The proposed evaluation metrics are suitable for multi-label hierarchical extreme classification scenarios. |
Peeking inside the black box: A Commonsense-aware Generative Framework for Explainable Complaint Detection (2023.acl-long)
Copied to clipboard
| Challenge: | Complaining is an expression of negative emotions communicated due to a discrepancy between reality and expectations. |
| Approach: | They propose to use an explainable complaint dataset to generate a commonsense-aware generative framework that can predict the complaint cause, severity level, emotion, and polarity of the text. |
| Outcome: | The proposed model can predict the complaint cause, severity level, emotion, and polarity of the text in addition to detecting whether it is a complaint or not. |
Revisiting the Evaluation of End-to-end Event Extraction (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing EE research uses the role-averaged evaluation metric, but it is misleading to downstream applications. |
| Approach: | They propose two new evaluation metrics that explicitly penalize wrongly identified event arguments. |
| Outcome: | The proposed evaluation metrics improve the initial evaluation by 10% . the proposed training scheme is better than the existing one, the authors show . |
Learning Only from Relevant Keywords and Unlabeled Documents (D19-1)
Copied to clipboard
| Challenge: | Existing methods for document classification are limited due to labeling and privacy concerns. |
| Approach: | They propose a super-vised text classification framework that provides keywords as a hint for classifying a document to a target class. |
| Outcome: | The proposed framework is simple to implement and has flexible choices of models, e.g., linear models or neural networks. |
RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework (2025.acl-long)
Copied to clipboard
Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing evaluation metrics for RAG systems are lacking due to high costs of data construction and lack of factual accuracy. |
| Approach: | They propose a framework to evaluate RAG systems in specialized scenarios . they propose three new metrics to evaluate LLM-generated responses . |
| Outcome: | The proposed framework outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. |
Towards a Better Metric for Evaluating Question Generation Systems (D18-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics based on n-gram similarity do not correlate well with human judgments . large datasets for document Question Answering (QA) have enabled the development of end-to-end supervised models . |
| Approach: | They propose a scoring function to capture answerability of questions . they also integrate existing similarity metrics into the scoring function . |
| Outcome: | The proposed scoring function improves human judgments on question answerability . the proposed scoring functions are made publicly available . |
Personalized Topic Selection Model for Topic-Grounded Dialogue (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing topic-grounded dialogue systems tend to predict user-uninteresting and contextually irrelevant topics due to noise within side information sources. |
| Approach: | They propose a personalized topic selection model for topic-grounded dialogue that selectively aggregates side information to generate engaging responses. |
| Outcome: | The proposed model outperforms state-of-the-art models on multiple evaluation metrics. |
Automatic, Meta and Human Evaluation for Multimodal Summarization with Multimodal Output (2024.naacl-long)
Copied to clipboard
| Challenge: | Multimodal summarization with multimodal output (MSMO) has attracted increasing research interest . evaluation is an emerging yet underexplored research topic . |
| Approach: | They propose a framework that studies three research questions of MSMO evaluation . they propose an automatic evaluation metric and a meta-evaluation benchmark dataset . |
| Outcome: | The proposed evaluation metric and human-annotated meta-evaluation benchmark are used to assess the quality of evaluation metrics and show the framework is effective. |
MUSCLE: A Model Update Strategy for Compatible LLM Evolution (2024.findings-emnlp)
Copied to clipboard
Jessica Echterhoff, Fartash Faghri, Raviteja Vemulapalli, Ting-Yao Hu, Chun-Liang Li, Oncel Tuzel, Hadi Pouransari
| Challenge: | Large Language Models (LLMs) are regularly updated to enhance performance, typically through changes in data or architecture. |
| Approach: | They propose a method to minimize the extent of instance regression in model updates by training a compatibility adapter that can enhance task fine-tuned language models. |
| Outcome: | The proposed approach reduces regressions by up to 40% when updating LLMs to newer versions while maintaining overall performance gains. |
Data-Efficient Concept Extraction from Pre-trained Language Models for Commonsense Explanation Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract concepts from pre-trained language models are not suitable for commonsense explanation generation. |
| Approach: | They propose a method to extract the key explanation concept from pre-trained language models by fine-tuning it with 20% training data and using a metric to evaluate the retrieved concepts. |
| Outcome: | The proposed method improves evaluation metrics over pre-trained language models and the existing models. |
Entity-aware Image Caption Generation (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning approaches generate generic descriptions of visual content and ignore background information. |
| Approach: | They propose a task which generates informative image captions using images and hashtags as input. |
| Outcome: | The proposed model outperforms unimodal baselines significantly with evaluation metrics on a dataset from Flickr. |
Query Refinement Prompts for Closed-Book Long-Form QA (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can answer questions and produce long-form texts, but the latter is difficult to evaluate since they are subjective in nature. |
| Approach: | They propose query refinement prompts that encourage LLMs to express multifacetedness and generate long-form answers covering multiple facets of the question. |
| Outcome: | The proposed model outperforms fully finetuned models in the closed-book setting and retrieve-then-generate open-book models. |
Multi-level Community-awareness Graph Neural Networks for Neural Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Recent studies have used Graph Neural Networks (GNNs) to encode language knowledge into token embeddings. |
| Approach: | They propose a multi-level community-awareness Graph Neural Network layer to jointly model local and global relationships between words and their linguistic roles in multiple communities. |
| Outcome: | The proposed method reduces time complexity in very long sentences while preserving the original meaning. |
Trust in Internal or External Knowledge? Generative Multi-Modal Entity Linking with Knowledge Retriever (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing generative approaches struggle with the knowledge gap between visual entity information and the intrinsic parametric knowledge of LLMs. |
| Approach: | They propose a knowledge retrieval method that leverages external sources to enhance visual entity information and a prioritization scheme that handles noisy retrieval results. |
| Outcome: | The proposed method shows improvements of 3.0% to 6.5% across all evaluation metrics compared to baselines. |
Joint Multitask Learning for Community Question Answering Using Task-Specific Embeddings (D18-1)
Copied to clipboard
| Challenge: | Stack-Overflow, Quora, and Yahoo! Answers forums are not moderated, which results in noisy and redundant content. |
| Approach: | They use deep neural networks to learn meaningful task-specific embeddings . they incorporate the embeddables into a conditional random field model . |
| Outcome: | The proposed task improves significantly across evaluation metrics. |
Hierarchical Catalogue Generation for Literature Review: A Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Scientific literature review generation aims to extract and organize important information from an abundant collection of reference papers and produces corresponding reviews while lacking a clear and logical hierarchy. |
| Approach: | They propose a task to generate a hierarchical catalogue of a review paper given various references by using a database of 7.6k literature review catalogues and 389k reference papers. |
| Outcome: | The proposed method produces a hierarchical catalogue of a review paper given various references. |
Enhancing Argument Summarization: Prioritizing Exhaustiveness in Key Point Generation and Introducing an Automatic Coverage Evaluation Metric (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for summarizing arguments are incapable of distinguishing between generated key points of different qualities. |
| Approach: | They propose an extractive approach that generates concise, high quality key points . they propose to use a clustering approach to generate key points from raw arguments . |
| Outcome: | The proposed method outperforms state-of-the-art methods for key point generation . it offers concise, high quality generated key points with higher coverage of reference summaries . |
Diversifying Question Generation over Knowledge Base via External Natural Questions (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods on knowledge base question generation focus on refining the quality of a single generated question. |
| Approach: | They propose a new diversity evaluation metric which measures the diversity among top-k generated questions for each instance while ensuring their relevance to the ground truth. |
| Outcome: | The proposed model outperforms pre-trained language model baselines and text-davinci-003 in diversity while achieving comparable performance with ChatGPT. |
EmoKnob: Enhance Voice Cloning with Fine-Grained Emotion Control (2024.emnlp-main)
Copied to clipboard
| Challenge: | EmoKnob framework allows fine-grained emotion control in speech synthesis with few-shot demonstrative samples of arbitrary emotion. |
| Approach: | They propose a framework that allows fine-grained emotion control in speech synthesis . they propose two methods to apply emotion control on emotions described by open-ended text . |
| Outcome: | The proposed framework allows fine-grained emotion control in speech synthesis with few-shot demonstrative samples of arbitrary emotion. |
DRBO: Mitigating Short Board Effect via Dynamic Reward Balancing in Multi-reward LLM Optimization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new framework to optimize large language models (LLMs) for evaluation metrics is needed to balance weaker metrics. |
| Approach: | They propose a Dynamic Reward Balancing Optimization framework to mitigate the "short-board effect" they apply it to single-task and multi-type task scenarios . |
| Outcome: | The proposed framework improves performance and balances performance across multiple metrics. |
SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | N-gram-based evaluation metrics are unreliable due to low correlation to human judgments. |
| Approach: | They propose a metric that rewards correct details and penalizes incorrect ones. |
| Outcome: | The proposed metric matches the performance of open-source LLM-based metrics in correlation to human judgments while being far more efficient. |
Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents (2024.acl-long)
Copied to clipboard
Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Liujianfeng Liujianfeng, Ang Li, Jian Luan, Bin Wang, Rui Yan, Shuo Shang
| Challenge: | Existing benchmarks for LLM-based mobile agents are insufficient to evaluate their capabilities. |
| Approach: | They propose a benchmark to evaluate LLM-based mobile agents' planning capabilities . they expand UI operations by incorporating 103 APIs to accelerate task completion . |
| Outcome: | The proposed benchmarks are based on 103 collected APIs and real user queries . the data is categorized into three distinct groups: SAST, SAMT, and MAMT . |
WSL-DS: Weakly Supervised Learning with Distant Supervision for Query Focused Multi-Document Abstractive Summarization (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to generate abstractive summarizations are lacking labeled training datasets. |
| Approach: | They propose a weakly supervised approach to generate a strong summary from a set of documents based on a query. |
| Outcome: | The proposed approach sets a new state-of-the-art in terms of evaluation metrics on the Document Understanding Conferences dataset. |
MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for memory-augmented dialogue systems lack practical value . current evaluation methods only consider passive memory retrieval while ignoring diverse memory recall with rich triggering factors. |
| Approach: | They propose to use long-term memory to create human-like dialogues using chatbots. |
| Outcome: | The proposed benchmark covers memory retrieval and memory recognition tasks with both passive and proactive memory recall data. |
TripTailor: A Real-World Benchmark for Personalized Travel Planning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for travel planning rely on unrealistic simulated data . fewer than 10% of the itineraries generated by the latest state-of-the-art LLMs achieve human-level performance. |
| Approach: | They propose a benchmark for personalized travel planning in real-world scenarios . they identify several critical challenges in travel planning including feasibility and rationality . |
| Outcome: | The proposed benchmarks show that fewer than 10% of the itineraries generated by the latest state-of-the-art LLMs achieve human-level performance. |
Synchronized Video Storytelling: Generating Video Narrations with Structured Storyline (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on dense video captioning and video story generation have made some progress, but in practical applications, we typically require synchronized narrations for ongoing visual scenes. |
| Approach: | They propose a task of Synchronized Video Storytelling to generate synchronized narrations for videos using a benchmark dataset with rich annotations. |
| Outcome: | The proposed framework can generate narrations with the guidance of the generated or predefined storyline and human evaluations validate the effectiveness. |
Scalable and Domain-General Abstractive Proposition Segmentation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Several recent studies have demonstrated the utility of proposition segmentation for downstream tasks. |
| Approach: | They propose a scalable, yet accurate, proposition segmentation model that can be supervised by LLMs. |
| Outcome: | The proposed model improves on training on annotated datasets and shows that it is easy to use. |
Gender Biases in Automatic Evaluation Metrics for Image Captioning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained evaluation metrics can perpetuate and amplify biases, causing inability to differentiate between biased and unbiased generations. |
| Approach: | They conduct a systematic study of gender biases in image captioning tasks . they show that pretrained models perpetuate and amplify biase . |
| Outcome: | The proposed model-based evaluation metrics have shown good correlations with human judgments in language generation tasks. |
LoRMA: Low-Rank Multiplicative Adaptation for LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have shown impressive generalization capabilities, but can be expensive to fine-tune due to high computational costs. |
| Approach: | They propose a low-rank multiplicative Adaptation technique that shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. |
| Outcome: | The proposed approach overcomes computational complexity and rank bottlenecks in terms of matrix multiplication metrics. |
QuestEval: Summarization Asks for Fact-based Evaluation (2021.emnlp-main)
Copied to clipboard
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, Patrick Gallinari
| Challenge: | Existing evaluation metrics for summarization evaluation are limited and do not correlate well with human judgments. |
| Approach: | They propose to extend existing evaluation metrics to include question answering models to assess whether a summary contains all relevant information in its source document. |
| Outcome: | The proposed framework significantly improves the correlation with human judgments over four evaluation dimensions. |
Offensive Video Detection: Dataset and Baseline Results (2020.lrec-1)
Copied to clipboard
| Challenge: | a large number of social media platforms discourage users from publishing offensive content . however, there is no method to detect offensive content on these platforms due to the high volume of publications. |
| Approach: | They propose to use text-based machine learning to detect offensive content on different platforms . they use word embedding with Deep Learning classifiers to perform best results . |
| Outcome: | The proposed methods outperform Classic and Deep Learning classifiers in Portuguese and CNN architectures in other features. |
VQA-Augmented Machine Translation with Cross-Modal Contrastive Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multimodal machine translation methods often extract visual features using pre-trained models while learning text features from scratch, leading to representation imbalance. |
| Approach: | They propose a cross-modal VQA-augmented multimodal machine translation method . it aligns image-source text pairs and image-question text pairs through dual-text contrastive learning . |
| Outcome: | The proposed method outperforms state-of-the-art methods on multiple evaluation metrics. |
DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. |
| Approach: | They propose a metric that evaluates natural language generation tasks as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models without training on evaluation datasets. |
| Outcome: | The proposed metric achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which exhibits strong dimension-level / task-level generalization ability and interpretability. |
A Sentiment Consolidation Framework for Meta-Review Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in abstractive text summarization have created plausible summaries, but it is unclear if they truly possess the capability of information consolidation to generate summary. |
| Approach: | They propose to prompt large language models to generate meta-reviews and use evaluation metrics to assess the quality of generated meta- reviews. |
| Outcome: | The proposed framework proves that human meta-reviewers follow a framework of sentiment consolidation to write meta- reviews compared with prompting them with simple instructions. |
Improving LLM Attributions with Randomized Path-Integration (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in AI research have impacted numerous application domains, fueling innovation and progress in user modeling and personalization. |
| Approach: | They propose a path-integration method for explaining language models via randomization of the integration path over the attention information in the model. |
| Outcome: | The proposed method outperforms state-of-the-art methods across 4 datasets and 5 evaluation metrics. |
SimOAP: Improve Coherence and Consistency in Persona-based Dialogue Generation via Over-sampling and Post-evaluation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work on large-scale corpora-based language models is limited and hard to generalize to all types of pre-trained language models. |
| Approach: | They propose a two-stage SimOAP strategy that over-samples and post-evaluates large-scale responses from existing models and selects a good response based on multiple evaluation metrics. |
| Outcome: | The proposed strategy outperforms baseline and automatic evaluation strategies in both automatic and human evaluations. |
Improving Neural Conversational Models with Entropy-Based Data Filtering (P19-1)
Copied to clipboard
| Challenge: | Current neural network-based conversational models lack diversity and generate boring responses to open-ended utterances. |
| Approach: | They propose an unsupervised method of filtering dialog datasets by removing generic utterances from training data using an entropy-based approach that does not require human supervision. |
| Outcome: | The proposed method improves dialog quality as chatbots learn to output more diverse responses to open-ended utterances. |
Evaluating Generative Language Models in Information Extraction as Subjective Question Correction (2024.lrec-main)
Copied to clipboard
| Challenge: | Modern large language models (LLMs) perform poorly in elementary tasks like relation extraction and event extraction due to two issues in conventional evaluation methods. |
| Approach: | They propose a method to evaluate large language models by incorporating a human annotation schema. |
| Outcome: | The proposed evaluation method improves matching between model outputs and golden labels. |
Evaluating Saliency Explanations in NLP by Crowdsourcing (2024.lrec-main)
Copied to clipboard
| Challenge: | a crowdsourced method to evaluate saliency methods in NLP is proposed . saliencies are difficult for humans to understand, and can cause psychological harm . |
| Approach: | They propose a method to evaluate saliency methods in NLP by crowdsourcing . they recruited 800 crowd workers and empirically evaluated seven salience methods . |
| Outcome: | The proposed method evaluates saliency methods on two datasets using crowdsourced data . it shows that the results are comparable to existing methods on NLP and CV fields . |
Understanding Faithfulness and Reasoning of Large Language Models on Plain Biomedical Summaries (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Generating plain biomedical summaries with Large Language Models (LLMs) can enhance access to biomedically knowledge. |
| Approach: | They propose a benchmark dataset with expert-annotated Faithfulness and Reasoning on plain biomedical summaries. |
| Outcome: | The proposed dataset shows that LLMs perform poorly in generating faithful biomedical summaries and that abstractiveness and faithfulness are negatively correlated. |
Evaluation of Really Good Grammatical Error Correction (2024.lrec-main)
Copied to clipboard
| Challenge: | emergence of large language models has highlighted the shortcomings of evaluation methods . evaluators often use grammatical error correction (GEC) to correct language errors at multiple levels . |
| Approach: | They perform a comprehensive evaluation of various GEC systems using Swedish learner texts . they suggest using human post-editing to analyze amount of change required to reach native-level human performance . |
| Outcome: | The proposed evaluations outperform existing methods for grammatical error correction in Swedish . the results highlight the shortcomings of existing evaluation methods . |
On the Correlation of Word Embedding Evaluation Metrics (2020.lrec-1)
Copied to clipboard
| Challenge: | Word embeddings are geometrical representations of word paradigmatics and syntagmatics. |
| Approach: | They propose to investigate evaluation metrics on various datasets to find correlations . they propose a fast solution to select the best word embeddings among many others . |
| Outcome: | The proposed method could be used to select the best word embeddings among many others. |
ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Automatically generated radiology reports often receive high scores from existing evaluation metrics but fail to earn clinicians’ trust. |
| Approach: | They propose a meta-evaluation framework that uses criteria spanning discrimination, robustness, and monotonicity to evaluate existing metrics. |
| Outcome: | The proposed framework offers guidance for building more clinically reliable evaluation methods. |
Compositional Generalization for Data-to-Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Data-to-text generation models can be used to generate textual descriptions from structured data . despite advances, systems struggle when confronted with unseen combinations of predicates . |
| Approach: | They propose a data-to-text generation model that addresses compositional generalization by clustering predicates into groups. |
| Outcome: | The proposed model outperforms T5-baselines in all evaluation metrics. |
Take a Break in the Middle: Investigating Subgoals towards Hierarchical Script Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing work assumes that events are sequentially arranged in a script, while this assumption leads to linear generation that is far from sufficient for comprehensively acquiring the representation about how events are organized towards a task goal. |
| Approach: | They propose to extend goal-oriented Script Generation task from the perspective of cognitive theory by incorporating subgoals into hierarchical script generation. |
| Outcome: | The proposed task is based on a new dataset and human evaluation metrics. |
Sentence Pair Embeddings Based Evaluation Metric for Abstractive and Extractive Summarization (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics, such as ROUGE and BLEU, rely on exact word matching and fail to capture semantic similarity. |
| Approach: | They propose to use contextualized word or sentence embeddings to capture semantic similarity between sentences to evaluate text summarization methods. |
| Outcome: | The proposed evaluation metric shows that it performs faster than the current state-of-the-art on the SummEval dataset. |
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports (P19-1)
Copied to clipboard
| Challenge: | Existing studies do not consider the complex structure information between and within report sections. |
| Approach: | They propose a framework which exploits the structure information between and within report sections for generating CXR imaging reports. |
| Outcome: | The proposed framework achieves state-of-the-art performance on two CXR report datasets. |
AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite (2023.emnlp-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation parsers have improved in recent years, but not solved. |
| Approach: | They propose an evaluation suite that evaluates AMR parsers on a range of phenomena . they find that current parser outputs are far from satisfactory . |
| Outcome: | The proposed evaluation suite reveals the abilities and shortcomings of current parsers. |
Improving Image Captioning with Better Use of Caption (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to image captioning focus on visual attention, but many do not. |
| Approach: | They propose a framework that explores semantics available in captions and leverages that to enhance both image representation and caption generation. |
| Outcome: | The proposed framework outperforms baselines on the MSCOCO dataset and is state-of-the-art under a wide range of evaluation metrics. |
Schema Generation for Large Knowledge Graphs Using Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Schemas are a vital part of ontology engineering and require substantial knowledge engineers and domain experts to create them. |
| Approach: | They propose to use large language models to generate schemas in Shape Expressions (ShEx) to bridge the resource gap between knowledge engineers and domain experts. |
| Outcome: | The proposed pipelines use local and global information from knowledge graphs (KGs) to generate high-quality schemas in Shape Expressions (ShEx). |
Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics are conflated and can mislead models, resulting in downstream harms. |
| Approach: | They propose a framework for conceptualizing and evaluating the reliability and validity of evaluation metrics based on empirical data. |
| Outcome: | The proposed framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. |
Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in various tasks, but may rely on dataset biases as shortcuts for prediction. |
| Approach: | They propose to use a test suite to evaluate the impact of shortcuts on LLMs' performance. |
| Outcome: | The proposed test suite incorporates six shortcut types, five evaluation metrics, and four prompting strategies. |
Why Does New Knowledge Create Messy Ripple Effects in LLMs? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing research has focused on post-training knowledge editing (KE) for language models to ensure that knowledge remains accurate and up-to-date. |
| Approach: | They propose to use a GradSim indicator to detect when and why updated knowledge ripples in language models. |
| Outcome: | The proposed indicator GradSim shows that LMs that fail to handle ripple effects have low GradSIM. |
HAUSER: Towards Holistic and Automatic Evaluation of Simile Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Similes are a crucial part of creative writing, but there is still a lack of evaluation metrics for simile generation. |
| Approach: | They propose to use similes as a tool to evaluate simile generation metrics . they propose to combine five criteria and automatic metrics for each criterion . |
| Outcome: | The proposed metrics are significantly more correlated with human ratings from each perspective compared with prior automatic metrics. |
MDPO: Customized Direct Preference Optimization with a Metric-based Sampler for Question and Answer Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods for QA data generation are limited by the dependence of existing evaluation metrics on ground truth labels. |
| Approach: | They propose a set of unsupervised evaluation metrics for QA data that enable multidimensional assessment based on the relationships among context,question and answer. |
| Outcome: | The proposed method outperforms state-of-the-art methods on public datasets and shows that it produces high-quality and domain-specific QA pairs. |
Evaluating the Knowledge Dependency of Questions (2022.emnlp-main)
Copied to clipboard
Hyeongdon Moon, Yoonseok Yang, Hangyeol Yu, Seunghyun Lee, Myeongho Jeong, Juneyoung Park, Jamin Shin, Minsam Kim, Seungtaek Choi
| Challenge: | Existing evaluation metrics for MCQ generation focus on the n-gram based similarity of the generated MCq to the gold sample and disregard their educational value. |
| Approach: | They propose to use a human survey to measure the MCQ’s answerability given knowledge of the target fact. |
| Outcome: | The proposed methods measure the MCQ’s answerability given knowledge of the target fact. |
CoSimLex: A Resource for Evaluating Graded Word Similarity in Context (2020.lrec-1)
Copied to clipboard
Carlos Santos Armendariz, Matthew Purver, Matej Ulčar, Senja Pollak, Nikola Ljubešić, Mark Granroth-Wilding
| Challenge: | Existing methods to evaluate word embeddings ignore context and treat words in isolation. |
| Approach: | They propose to build a new word embeddings-based dataset that provides context-dependent similarity measures. |
| Outcome: | The proposed dataset provides context-dependent similarity measures and covers a well-resourced language (English) but a number of less-resource languages. |
SUMIE: A Synthetic Benchmark for Incremental Entity Summarization (2025.coling-main)
Copied to clipboard
| Challenge: | Existing datasets that test incrementally update entity summaries are lacking. |
| Approach: | They propose a fully synthetic dataset that exposes real-world IES challenges by generating diverse attributes, summaries, and unstructured paragraphs with 99% alignment accuracy. |
| Outcome: | The proposed dataset shows that state-of-the-art LLMs struggle to update summaries with an F1 higher than 80.4%. |
Models See Hallucinations: Evaluating the Factuality in Video Captioning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics show little correlation with human factuality annotation. |
| Approach: | They propose a weakly-supervised, model-based factuality metric FactVC which outperforms previous metrics on factual evaluation of video captioning. |
| Outcome: | The proposed model outperforms previous metrics on factuality evaluation of video captioning. |
X-ACE: Explainable and Multi-factor Audio Captioning Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for automated audio captioning only provide an overall score . current evaluation checklists are inadequate to characterize the nuanced differences . |
| Approach: | They propose an explainable and multi-factor audio captioning evaluation paradigm . they define sound event, source, attribute and relation as four factors tailored for the audio description . |
| Outcome: | The proposed evaluation paradigm improves the quality of audio captions . it can detect mismatches and align with human perception, the authors show . |
Learning Personalized Alignment for Evaluating Open-ended Text Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Traditional evaluation metrics rely heavily on lexical similarity with human-written references, showing poor correlation with human judgments and failing to account for alignment with the diversity of human preferences. |
| Approach: | They propose an interpretable evaluation framework that evaluates alignment with specific human preferences by providing detailed comments and fine-grained scoring. |
| Outcome: | The proposed framework outperforms GPT-4 in Kendall correlation and accuracy with zero-shot reviewers. |
LLM-Enhanced Query Generation and Retrieval Preservation for Task-Oriented Dialogue (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge retrieval methods for task-oriented dialogues are limited by data scarcity and lack of data to annotate. |
| Approach: | They propose an LLM-enhanced model of query-guided knowledge retrieval for task-oriented dialogue . they propose to select the most relevant knowledge from retrieved top-K records and incorporate them as prompts to guide a generator in response generation. |
| Outcome: | The proposed model outperforms state-of-the-art in three benchmarks on three standard benchmarks. |
Simplicity Level Estimate (SLE): A Learned Reference-Less Metric for Sentence Simplification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics conflate simplicity with correlated attributes such as fluency or meaning preservation. |
| Approach: | They propose a new learning evaluation metric that focuses on simplicity outperforming most existing metrics in terms of correlation with human judgements. |
| Outcome: | The proposed metric outperforms most existing metrics in terms of correlation with human judgements. |
Re-evaluating Evaluation in Text Summarization (2020.emnlp-main)
Copied to clipboard
| Challenge: | Automated evaluation metrics are an essential part of the development of text-generation tasks such as summarization. |
| Approach: | They propose to use top-scoring system outputs to assess the reliability of automatic evaluation metrics for text summarization. |
| Outcome: | The proposed evaluation method is based on human judgments from 25 top-scoring neural summarization systems. |
On the Limitations of Reference-Free Evaluations of Generated Text (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study has shown that evaluation metrics which accurately estimate the quality of generated text are limited in their ability to evaluate generated text. |
| Approach: | They argue that reference-free metrics are limited in their ability to evaluate generated text . they recommend that they be used as diagnostic tools for analyzing and understanding model behavior . |
| Outcome: | The proposed evaluation metrics are limited in their ability to evaluate generated text . they can be optimized at test time, can be biased against models with similar outputs . |
Benchmarking Machine Translation with Cultural Awareness (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on terminology translation focus on breaking language barriers rather than cultural barriers. |
| Approach: | They propose a parallel corpus enriched with CSI annotations in 6 language pairs for investigating Cultural-Aware Machine Translation. |
| Outcome: | The proposed corpus is enriched with CSI annotations in 6 languages and measures translation quality. |
Faster Minimum Bayes Risk Decoding with Confidence-based Pruning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Minimum Bayes risk (MBR) decoding is a decision rule for conditional sequence generation tasks. |
| Approach: | They propose an algorithm which grows the number of samples used to estimate utility . it prunes hypotheses that are unlikely to have the highest utility based on bootstrap sampling . |
| Outcome: | The proposed method outperforms beam search in conditional language generation and neural machine translation tasks while being statistically indistinguishable from other proposed methods. |
Asking Clarification Questions to Handle Ambiguity in Open-Domain QA (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Ambiguous questions persist in open-domain question answering because formulating a precise question with a unique answer is often challenging. |
| Approach: | They propose to ask a clarification question where the user’s response will help identify the interpretation that best aligns with the user's intention. |
| Outcome: | The proposed approach achieves F1 of 61.3, 25.1, and 40.5 on the three tasks, demonstrating the need for further improvements while providing competitive baselines for future work. |
OntologyRAG-Q: Resource Development and Benchmarking for Retrieval-Augmented Question Answering in Qur’anic Tafsir (2025.emnlp-main)
Copied to clipboard
| Challenge: | An annotated Tafsir ontology and a collection of 15 structured Tafsian books are presented in this paper. |
| Approach: | They propose a framework for retrieval and question-answering Tafsir data that spans the entire pipeline from dataset construction through evaluation and error analysis. |
| Outcome: | The proposed framework achieves 69.52% accuracy and 74.36% correctness overall, though multi-hop and context-dependent questions remain challenging. |
Do You Hear The People Sing? Key Point Analysis via Iterative Clustering and Abstractive Summarisation (2023.acl-long)
Copied to clipboard
| Challenge: | Argument summarisation is a promising but currently under-explored field. |
| Approach: | They propose a framework to generate key points from short texts in a task known as Key Point Analysis. |
| Outcome: | The proposed framework improves state-of-the-art in argument summarisation with performance improvement of 14 percentage points compared to ROUGE and human evaluation scores. |
Evaluating Factuality in Cross-lingual Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for monolingual summarization require translation to evaluate the factuality of cross-lingual summmarization. |
| Approach: | They propose to analyze cross-lingual factuality by collecting annotations and generated summaries from models at summary level and sentence level. |
| Outcome: | The proposed dataset shows that over 50% of generated summaries contain factual errors with different characteristics from monolingual summarization. |
Self-Awareness before Action: Mitigating Logical Inertia via Proactive Cognitive Awareness (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing work on abductive and long-context reasoning reports that current models still lack self-awareness of missing premises. |
| Approach: | They propose a reasoning framework that introduces self-awareness of missing premises before making the final decision. |
| Outcome: | SABA achieves best performance on all three difficulty splits of detective puzzle benchmark . a small early mistake can remain uncorrected and can guide later reasoning . |
A Unified View of Evaluation Metrics for Structured Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | a framework that unifies evaluation metrics for structured prediction tasks is presented . metric design decisions are motivated by specific characteristics of tasks, and we suggest modifications to existing metrics to meet those motivations. |
| Approach: | They propose a framework that unifies a variety of evaluation metrics for different structured prediction tasks. |
| Outcome: | The proposed framework can be used to create new metrics based on the output structure of a number of tasks. |
Python Code Generation by Asking Clarification Questions (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work addresses text-to-code generation using pretrained language models (PLMs) for large-scale NLD: Logistic Regression. |
| Approach: | They propose a dataset containing pairs of natural language descriptions and code with created synthetic clarification questions and answers to solve the under-specified nature of a natural language description. |
| Outcome: | The proposed model improves on previous models, while introducing new challenges to the community, including when and what clarification questions should be asked. |
Regression Aware Inference with LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong results on a range of applications, including regression and scoring tasks. |
| Approach: | They propose alternative inference strategies that estimate the Bayes-optimal solution for regression and scoring metrics in closed-form from sampled responses. |
| Outcome: | The proposed approach significantly improves over baselines across datasets and models. |
Contrastive Conditioning for Assessing Disambiguation in MT: A Case Study of Distilled Bias (2021.emnlp-main)
Copied to clipboard
| Challenge: | Lexical disambiguation is a major challenge for machine translation systems . previous work focused on automatic post-hoc analysis of translations, but rules of what makes a disambiguations correct or incorrect tend to be imprecise. |
| Approach: | They propose a black-box method that uses contrastive conditioning to detect disambiguation errors. |
| Outcome: | The proposed method is scalable and reliable for disambiguation evaluations. |
A Survey of Cognitive Distortion Detection and Classification in NLP (2025.findings-emnlp)
Copied to clipboard
| Challenge: | despite momentum in natural language processing, the field remains fragmented . inconsistencies in CD taxonomies, task formulations and evaluation practices limit comparability . |
| Approach: | This review provides a comprehensive review of 38 studies spanning two decades . they map how CDs have been implemented in computational research and evaluate the methods applied. |
| Outcome: | The paper presents the first comprehensive review of 38 studies spanning two decades . it summarises common task setups and highlights persistent challenges to support more coherent research. |
MultiSQL: A Schema-Integrated Context-Dependent Text2SQL Dataset with Diverse SQL Operations (2024.findings-acl)
Copied to clipboard
| Challenge: | Text2SQL is a task that translates natural language into SQL statements. |
| Approach: | They propose a task that translates natural language into SQL statements. |
| Outcome: | The proposed task enables users to convert natural language into SQL statements. |
Would LLMs be Good Historical Linguists and Chinese Dialect Learners? (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models struggle with low-resource Chinese dialects due to substantial phonological divergence. |
| Approach: | They propose to incorporate Middle Chinese, the common historical ancestor of modern Chinese dialects, into LLMs to improve dialectal pronunciation modeling. |
| Outcome: | The proposed approach improves on standard Chinese but struggles with low-resource Chinese dialects . the proposed model improves over baselines while revealing variation across dialects. |
TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planning (2025.acl-long)
Copied to clipboard
Soumyabrata Chaudhuri, Pranav Purkar, Ritwik Raghav, Shubhojit Mallick, Manish Gupta, Abhik Jana, Shreya Ghosh
| Challenge: | Existing benchmarks such as TravelPlanner and TravelPlann+ rely on semi-synthetic data and lack key real-world constraints. |
| Approach: | They propose a spatio-temporally coherent travel planning dataset incorporating real-world constraints, including public transit schedules, public events, varied attraction categories, and user personas for enhanced personalization. |
| Outcome: | The proposed dataset significantly improves meal scheduling, improving performance from 61% to 80% in the 7-day scenario. |
KazParC: Kazakh Parallel Corpus for Machine Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Statistical machine translation gained ground over rule-based machine translation in the late 1990s thanks to its ability to learn from large bilingual corpora. |
| Approach: | They propose to develop a parallel corpus for machine translation across Kazakh, English, Russian, and Turkish. |
| Outcome: | The proposed model outperforms Google Translate and Yandex Translate in terms of performance and evaluation metrics. |
It’s Not Bragging If You Can Back It Up: Can LLMs Understand Braggings? (2025.acl-long)
Copied to clipboard
| Challenge: | Bragging is a pervasive social-linguistic phenomenon that reflects complex human interaction patterns. |
| Approach: | They propose to use bragging recognition, bragging explanation, and bragging generation tasks to examine bragging in large language models (LLMs) . |
| Outcome: | The proposed models can identify bragging intent, social appropriateness, and account for context sensitivity and provide new insights into how LLMs process bragging. |
On Degrees of Freedom in Defining and Testing Natural Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale pretrained language models are performing increasingly well at various tasks and offering real-world applications. |
| Approach: | They propose to define NLU as an inductive evidence that the test subject understands the language sufficiently well to meet stakeholder objectives. |
| Outcome: | The proposed framework can be used to design credible tests and facilitate scientific communication. |
Sparsity-Accelerated Training for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated proficiency across various NLP tasks but often require additional training, such as continual pre-training and supervised fine-tuning. |
| Approach: | They propose to leverage sparsity in pre-trained LLMs to accelerate training by disregarding computations for unimportant neurons. |
| Outcome: | The proposed framework achieves comparable or superior performance to standard training while significantly accelerating the process. |
What’s Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing GUI reasoning methods rely on direct screen-based decision-making, which lacks interpretability and overlooks a comprehensive understanding of UI elements, ultimately leading to task failure. |
| Approach: | They propose a GUI reasoning paradigm that treats the GUI reasoning task as a cyclic ***Screen-UI elements-Action** process. |
| Outcome: | The proposed paradigm achieves state-of-the-art UI understanding performance while yielding superior results in GUI reasoning tasks. |
Are Your LLMs Capable of Stable Reasoning? (2025.findings-acl)
Copied to clipboard
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, Kai Chen
| Challenge: | Existing evaluation protocols and metrics do not capture the full spectrum of LLM capabilities, especially in complex reasoning tasks. |
| Approach: | They propose a new evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model’s potential capabilities and operational consistency. |
| Outcome: | The proposed evaluation metric measures model performance across multiple sampling attempts and provides comprehensive insights into their potential capabilities and operational consistency. |
Knowledge-Centric Templatic Views of Documents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on document generation has tackled the creation of each separate format as a different task, leading to fragmented learning processes, redundancy in models and methods, and disjointed evaluation. |
| Approach: | They propose a method that unifies the generation and evaluation of templatic views of documents in multiple formats. |
| Outcome: | The proposed method improves performance for heterogeneous downstream applications while reducing the need for task specific evaluation metrics. |
Reliable Evaluation and Benchmarks for Statement Autoformalization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating statement autoformalization are limited . current methods can achieve up to 45.1% accuracy on undergraduate mathematics . |
| Approach: | They propose a new autoformalization metric that correlates strongly with human judgment . they propose two new auto-formalisation benchmarks: ProofNet# and RLM25 . |
| Outcome: | The proposed methods can achieve up to 45.1% accuracy on undergraduate mathematics but struggle with research-level content without proper context. |
HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current benchmarks for Hebrew Natural Language Processing (NLP) focus mainly on morpho-syntactic tasks, neglecting the semantic dimension of language understanding. |
| Approach: | They propose to use Hebrew machine reading comprehension (MRC) as extractive Question Answering to address this problem. |
| Outcome: | The proposed benchmark features 30,147 question-answer pairs derived from both Hebrew Wikipedia articles and Israeli tech news. |
Transparentize the Internal and External Knowledge Utilization in LLMs with Trustworthy Citation (2025.findings-acl)
Copied to clipboard
| Challenge: | citation generation and retrieval-augmented generation are still lacking in large language models due to hallucinations. |
| Approach: | They propose a retrieval-augmented citation generation task that requires models to generate citations considering both external and internal knowledge while providing trustworthy references. |
| Outcome: | The proposed method achieves better performance across scenarios compared to baselines . retrieval quality, question types, and model knowledge influence trustworthiness . |
Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization (2025.acl-long)
Copied to clipboard
| Challenge: | n-gram-based metrics are considered indicative (even if imperfect) of human evaluation for English, but their suitability for other languages remains unclear. |
| Approach: | They systematically assess evaluation metrics for generation for languages and tasks using n-gram-based and neural-based metrics. |
| Outcome: | The proposed evaluation suite is based on eight languages from four typological families and shows that it is sensitivity to the language type at hand. |
Language Reconstruction with Brain Predictive Coding from fMRI Data (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that the perception of speech can be decoded from brain signals and subsequently reconstructed as continuous language. |
| Approach: | They propose to use FMRI-to-text decoding with Predictive coding to generate a main network and a side network to generate brain predictive representations from related regions of interest. |
| Outcome: | The proposed model outperforms current decoding models on several evaluation metrics on two naturalistic language comprehension fMRI datasets. |
Meta-Evaluation of Sentence Simplification Metrics (2024.lrec-main)
Copied to clipboard
| Challenge: | Automatic Text Simplification (ATS) is a major natural language processing task that aims to help people understand complex text. |
| Approach: | They propose to use a human-annotated dataset to study automatic text simplification models to determine which metrics to use when evaluating new models. |
| Outcome: | The proposed models reconstruct the text into a simpler format by deletion, substitution, addition or splitting, while preserving the original meaning and correct grammar. |
EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios (2026.acl-long)
Copied to clipboard
Bin Xu, Yu Bai, Huashan Sun, Yiguan Lin, Siming Liu, Xinyue Liang, Yaolin Li, Zhuangzhi Dong, Jingren Zhang, Yufan Deng, Xinyu Zou, Yang Gao, Heyan Huang
| Challenge: | Existing benchmarks that focus on knowledge-intensive tasks do not reflect diverse educational scenarios. |
| Approach: | They propose a benchmark that incorporates 9 major scenarios and 4,000 educational contexts. |
| Outcome: | The proposed model performs comparable to state-of-the-art large models on the test set. |
LLMTaxo: Leveraging Large Language Models for Constructing Taxonomy of Factual Claims from Social Media (2025.findings-acl)
Copied to clipboard
| Challenge: | Social media's global reach and ease of use have transformed how millions of users exchange opinions, news, and factual claims in real-time, making it fertile ground for misinformation. |
| Approach: | They propose a framework that leverages large language models to construct taxonomies of factual claims from social media by generating topics at multiple levels of granularity. |
| Outcome: | The proposed framework produces clear, coherent, and comprehensive taxonomies on three diverse datasets and outperforms other frameworks in most metrics. |
DocMEdit: Towards Document-Level Model Editing (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models only output short phrases or sentences, raising doubts about their practical usability. |
| Approach: | They propose a dataset focused on document-level model editing that aims to correct errors and outdated knowledge in Large language models (LLMs) they propose to use document-based model editing to improve model capabilities in real-world scenarios. |
| Outcome: | The proposed model editing task improves model capabilities in real-world scenarios and reduces the cost of retraining. |
DLTKG: Denoising Logic-based Temporal Knowledge Graph Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to temporal knowledge representation face limited generalization to unseen facts and insufficient interpretability of reasoning processes. |
| Approach: | They propose a framework that uses a denoising diffusion process to complete reasoning tasks . they propose introducing a noise source and historical conditionguiding mechanism to improve interpretability . |
| Outcome: | The proposed framework outperforms state-of-the-art methods on three benchmark datasets. |
Multilingual Coreference Resolution in Low-resource South Asian Languages (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing coreference resolution models for South Asian languages are limited . a a sanity check for the prediction of translations is required to ensure accuracy of the model, authors say . |
| Approach: | They evaluate an end-to-end coreference resolution model on a Hindi golden set . they use translation and word-alignment tools to translate a translated dataset into 31 languages . |
| Outcome: | The proposed model scored 64 and 68 on a Hindi golden set. |
DocCGen: Document-based Controlled Code Generation (2024.emnlp-main)
Copied to clipboard
Sameer Pimparkhede, Mehant Kammakomati, Srikanth Tamilselvam, Prince Kumar, Ashok Kumar, Pushpak Bhattacharyya
| Challenge: | Large language models (LLMs) produce state-of-the-art performance on natural language to code generation for resource-rich general-purpose languages like C++, Java, and Python. |
| Approach: | They propose a framework that breaks the NL-to-Code generation task into two steps . they use library documentation to detect the correct libraries and schema rules extracted from the documentation to constrain the decoding . |
| Outcome: | The proposed framework improves different sized language models across all six evaluation metrics, reducing syntactic and semantic errors in structured code. |
LLMs Can Achieve High-quality Simultaneous Machine Translation as Efficiently as Offline (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models perform well in offline machine translation when the complete source sentence is provided . however, in many real scenarios, the source tokens arrive in a streaming manner and simultaneous machine translation is required . |
| Approach: | They propose a new paradigm that includes constructing supervised fine-tuning data for simultaneous machine translation (SiMT) to achieve SiMT, source and target tokens are rearranged into interleaved sequences, separated by special tokens according to varying latency requirements. |
| Outcome: | The proposed approach achieves state-of-the-art performance across various SiMT benchmarks and evaluation metrics while maintaining efficient auto-regressive decoding. |
Neural Multimodal Topic Modeling: A Comprehensive Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural topic models can find coherent and diverse topics in textual data, but they are limited in dealing with multimodal datasets. |
| Approach: | They propose two new topic modeling solutions and two new evaluation metrics for document multimodality. |
| Outcome: | The proposed models generate coherent and diverse topics on a rich dataset. |
Stop Playing the Guessing Game! Evaluating Conversational Recommender Systems via Target-free User Simulation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | despite advances in CRSs, reliably assessing their ability to elicit preferences remains a challenge. |
| Approach: | They propose a user-CRS evaluation protocol with target-free user simulators . they show that current evaluation metrics emphasize single-turn recall of target items . |
| Outcome: | The proposed evaluation protocol is based on a simulation-based evaluation environment. |
Assessing LLM Reasoning Steps via Principal Knowledge Grounding (2025.findings-emnlp)
Copied to clipboard
Hyeon Hwang, Yewon Cho, Chanwoong Yoon, Yein Park, Minju Song, Kyungjae Lee, Gangwoo Kim, Jaewoo Kang
| Challenge: | Step-by-step reasoning has become a standard approach for large language models to tackle complex tasks. |
| Approach: | They propose a framework that assesses the knowledge grounding of intermediate reasoning by using a large-scale repository of atomic knowledge essential for reasoning. |
| Outcome: | The evaluation suite identifies missing or misapplied knowledge elements and provides crucial insights for uncovering fundamental reasoning deficiencies in LLMs. |
InsideOut: Measuring and Mitigating Insider–Outsider Bias in Interview Script Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Recent research has raised concerns about culture-related fairness issues in LLM-generated content. |
| Approach: | They propose to use 4,000 generation prompts and three evaluation metrics to quantify LLMs' **insider-outsider bias** . |
| Outcome: | The proposed method reduces bias in Llama model by 89.70% and mitigates bias on Qwen by 82.54% on cultural alignment gap metric. |
Emphasising Structured Information: Integrating Abstract Meaning Representation into LLMs for Enhanced Open-Domain Dialogue Evaluation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation metrics struggle to evaluate adversarial negative examples . existing metrics struggle in handling adversarials, resulting in low correlations with human judgments. |
| Approach: | They propose a framework that integrates AMR and domain-specific language models for automatic open-domain dialogue evaluation. |
| Outcome: | The proposed evaluation framework achieves strong correlations with human judgments across multiple datasets. |
RD-MCSA: A Multi-Class Sentiment Analysis Approach Integrating In-Context Classification Rationales and Demonstrations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multi-class sentiment analysis (MCSA) are difficult due to subtle semantic differences between adjacent sentiment levels and the scarcity of high-quality annotated data. |
| Approach: | They propose a framework to integrate classification rationales with adaptively selected demonstrations to enhance MCSA performance under limited supervision. |
| Outcome: | The proposed framework outperforms baseline and standard ICL methods on five benchmark datasets. |
Mining the uncertainty patterns of humans and models in the annotation of moral foundations and human values (2025.acl-long)
Copied to clipboard
| Challenge: | disagreement in annotation (HLV) is considered a constitutive feature of subjective tasks. |
| Approach: | They investigate the relationship between disagreement in annotation and model uncertainty . they use linguistic features to calibrate models to HLV and uncertainty to analyze their impact on uncertainty. |
| Outcome: | The proposed model uncertainty is calibrated to human label variation (HLV) the proposed model is calibrate to human labels, the authors show . |
CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics (2025.findings-emnlp)
Copied to clipboard
Shravan Nayak, Mehar Bhatia, Xiaofeng Zhang, Verena Rieser, Lisa Anne Hendricks, Sjoerd Van Steenkiste, Yash Goyal, Karolina Stanczak, Aishwarya Agrawal
| Challenge: | CulturalFrames is a benchmark designed for rigorous human evaluation of cultural representation in visual generations. |
| Approach: | They propose to quantify the alignment of T2I models and evaluation metrics with respect to both explicit (stated) and implicit (unstated, implied by the prompt’s cultural context) cultural expectations. |
| Outcome: | The proposed model is based on 983 prompts, 3637 images and 10k human annotations from 10 countries and 5 socio-cultural domains. |
T2A-Feedback: Improving Basic Capabilities of Text-to-Audio Generation via Fine-grained AI Feedback (2025.acl-long)
Copied to clipboard
Zehan Wang, Ke Lei, Chen Zhu, Jiawei Huang, Sashuai Zhou, Luping Liu, Xize Cheng, Shengpeng Ji, Zhenhui Ye, Tao Jin, Zhou Zhao
| Challenge: | Text-to-audio (T2A) models still struggle to satisfy human preferences for prompt-following and acoustic quality when generating complex multi-event audio. |
| Approach: | They propose to use AI feedback learning to enhance basic capabilities of text-to-audio models . they use a large audio preference dataset to evaluate the model's capabilities . |
| Outcome: | The proposed model improves in simple and complex scenarios with AI feedback learning. |
Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices. |
| Approach: | They propose an interpretable evaluation framework that evaluates MT metrics in two scenarios that serve as proxies for filtering and translation re-ranking use cases. |
| Outcome: | The proposed framework offers clearer insights than correlation with human judgments. |
Evaluating the Evaluation of Diversity in Commonsense Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for commonsense generation are unclear on which metrics are best suited for evaluating the diversity of outputs. |
| Approach: | They propose to use a large language model to analyze commonsense generation data to determine which diversity metrics are best suited for commonsensing. |
| Outcome: | The proposed metrics outperform form-based metrics and show high correlations with the LLM-based ratings. |
How do Language Models Reshape Entity Alignment? A Survey of LM-Driven EA Methods: Advances, Benchmarks, and Future (2025.emnlp-main)
Copied to clipboard
| Challenge: | Entity alignment (EA) is critical for knowledge graph (KG) integration. |
| Approach: | They propose a taxonomy that categorizes methods in three stages: data preparation, feature embedding, and alignment. |
| Outcome: | The proposed taxonomy categorizes methods in three key stages: data preparation, feature embedding, and alignment. |
Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluations (2024.lrec-main)
Copied to clipboard
| Challenge: | Personalization is a multifaceted process that requires multiple definitions and varies between individuals. |
| Approach: | They propose to systemically survey the recent landscape of personalized dialogue generation including the datasets employed, methodologies developed, and evaluation metrics applied. |
| Outcome: | The proposed model can generate fluent and coherent responses to human queries in a language-based conversational agent. |
Personalized Generation In Large Model Era: A Survey (2025.acl-long)
Copied to clipboard
Yiyan Xu, Jinghao Zhang, Alireza Salemi, Xinting Hu, Wenjie Wang, Fuli Feng, Hamed Zamani, Xiangnan He, Tat-Seng Chua
| Challenge: | Recent advances in large generative models have catalyzed a paradigm shift in content generation to Personalized Generation (PGen). |
| Approach: | They propose a multi-level taxonomy that systematically formalizes PGen's key components, core objectives, and abstract workflows. |
| Outcome: | The proposed taxonomy bridging PGen research across multiple modalities highlights open challenges and promising directions for future exploration. |
Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable performance on question-answering tasks due to their superior capabilities in natural language understanding and generation. |
| Approach: | They propose a structured taxonomy that categorizes the methodology of synthesizing LLMs and knowledge graphs for QA according to the categories of QA and the KG’s role when integrating with LLM. |
| Outcome: | The proposed taxonomy categorizes the methods according to the categories of QA and the KG’s role when integrating with LLMs. |
Scansion-based Lyrics Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | a new method for generating lyrics for Mandarin songs is based on scansion . the number of syllables required is variable due to the number and number of notes . |
| Approach: | They propose a method to generate Mandarin lyrics with a good match between melody and tonal contour. |
| Outcome: | The proposed system outperforms all other systems in lyric-melody fit and uses proxies for quantifying creativity. |
Semantics-aware prompting for translating NOtices To AirMen (2025.findings-acl)
Copied to clipboard
| Challenge: | A NOTAM or NOtice To AirMen is a crucial notification for different stakeholders . writing and understanding these messages puts heavy cognitive load on its end users. |
| Approach: | They propose a framework to translate NOTAMs into English natural language using LLMs . they use context-aware semantic prompting techniques paired with domain-specific rules . |
| Outcome: | The proposed approach improves translation accuracy and clarity using semantic prompting techniques . the proposed framework was evaluated using LLMs of varying sizes and 8 evaluation metrics . |
Don’t Sweat the Small Stuff: Segment-Level Meta-Evaluation Based on Pairwise Difference Correlation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to calculate agreement are acceq and acc eq, but there are many ways to compute agreement based on how scores are grouped together. |
| Approach: | They propose a segment-level meta-evaluation metric that utilizes pairwise differences rather than raw scores to refine Global Pearson to intra-segment comparisons. |
| Outcome: | The proposed metric correctly ranks sentinel evaluation metrics and better aligns with human error weightings than acceq. |
Automated Structured Radiology Report Generation (2025.acl-long)
Copied to clipboard
Jean-Benoit Delbrouck, Justin Xu, Johannes Moll, Alois Thomas, Zhihong Chen, Sophie Ostmeier, Asfandyar Azhar, Kelvin Zhenghao Li, Andrew Johnston, Christian Bluethgen, Eduardo Pontes Reis, Mohamed S Muneer, Maya Varma, Curtis Langlotz
| Challenge: | Existing models struggle to produce consistent, clinically meaningful reports and standard evaluation metrics fail to capture the nuances of radiological interpretation. |
| Approach: | They propose to reformulate free-text radiology reports into a standardized format, ensuring clarity, consistency, and structured clinical reporting. |
| Outcome: | The proposed task reformulates free-text radiology reports into a standardized format, ensuring clarity, consistency, and structured clinical reporting. |
Causal Tree Extraction from Medical Case Reports: A Novel Task for Experts-like Text Comprehension (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to extract causal relationships from medical case reports are insufficient for capturing causal relationships of an entire case. |
| Approach: | They propose a task that generates a causal tree with the primary disease as the root and extracts causal relationships from a medical case report. |
| Outcome: | The proposed method outperforms the baseline method by 20.2 points in the human evaluation and introduces evaluation metrics that reflect clinician preferences. |
A Dual-Perspective NLG Meta-Evaluation Framework with Automatic Benchmark and Better Interpretability (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics are insufficient to meet requirements for natural language generation. |
| Approach: | They propose a dual-perspective NLG meta-evaluation framework that focuses on different evaluation capabilities and a method of automatically constructing benchmarks without requiring new human annotations. |
| Outcome: | The proposed framework improves interpretability and provides better performance for 16 representative LLMs. |
InspireDebate: Multi-Dimensional Subjective-Objective Evaluation-Guided Reasoning and Optimization for Debating (2025.acl-long)
Copied to clipboard
| Challenge: | Existing LLMs focus on responding to specific arguments while neglecting objective assessments such as authenticity and logical validity. |
| Approach: | They propose a multi-dimensional evaluation system and an optimized debating framework . they propose to use coT reasoning enhancement, web-based Retrieval Augmented Generation to optimize across various dimensions. |
| Outcome: | The proposed framework outperforms baseline models in argument quality assessment and debate process simulation by 57%. |
TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for visual storytelling ignore latent topic information. |
| Approach: | They propose a topic-aware reinforcement network for VIsual StoryTelling that takes topic information into account to generate a coherent story. |
| Outcome: | The proposed method outperforms most of the competing models across multiple evaluation metrics. |
Rethinking Prompt-based Debiasing in Large Language Model (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing prompt-based methods for debiasing are often superficial and lack a thorough understanding of complex bias concepts. |
| Approach: | They analyze a BBQ and stereoSet benchmarks to examine the assumption that large language models understand biases. |
| Outcome: | The proposed model misclassified 90% of unbiased content as biased despite high accuracy on BBQ dataset . the proposed model may have been flawed in previous attempts to debiase . |
EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on explainable evaluation metrics generate explanations without standardized criteria and the overall quality of the generated explanations remains unverified. |
| Approach: | They propose a reference-free evaluation metric that provides structured explanations based on fluency, relevance, and descriptiveness. |
| Outcome: | The proposed evaluation template achieves state-of-the-art on benchmark datasets while providing significantly higher-quality explanations than existing metrics. |
When Words Smile: Generating Diverse Emotional Facial Expressions from Text (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems that generate only coarse facial expressions ignore the rich and dynamic nature of face-to-face communication. |
| Approach: | They propose an end-to-end text-to expression model that explicitly focuses on emotional dynamics. |
| Outcome: | The proposed model outperforms baselines on 15,000 text–3D expression pairs on a large-scale dataset. |
Social Bias in Multilingual Language Models: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained multilingual models exhibit the same social bias as models processing English texts. |
| Approach: | They examine the literature on bias evaluation and mitigation approaches in multilingual and non-English contexts and identify gaps in the field. |
| Outcome: | The proposed models perform well on multilingual language understanding benchmarks and are consistent with the current literature. |
TaxoAdapt: Aligning LLM-Based Multidimensional Taxonomy Construction to Evolving Research Corpora (2025.acl-long)
Copied to clipboard
| Challenge: | Recent automated taxonomies over-rely on a specific corpus, sacrificing generalizability, or depend heavily on the general knowledge of large language models (LLMs) . |
| Approach: | They propose a framework that dynamically adapts an LLM-generated taxonomy to a given corpus across multiple dimensions. |
| Outcome: | The proposed framework performs iterative hierarchical classification, expanding both the taxonomy width and depth based on corpus’ topical distribution. |
LiTransProQA: An LLM-based Literary Translation Evaluation Metric with Professional Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for literature prioritize mechanical accuracy over artistic expression . this bias could result in an irreversible decline in translation quality and cultural authenticity . |
| Approach: | They propose a novel, reference-free, LLM-based question-answering framework for literary translation evaluation. |
| Outcome: | a novel, reference-free, LLM-based question-answering framework is developed for literary translation evaluation. |
CogEmp:A Cognitive Empathy-Oriented Dialogue System for Structured Psychological Counseling (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing models lack accurate modeling of cognitive empathy, especially the ability to understand users’ emotions and their underlying psychological causes. |
| Approach: | They propose a model tailored for the Chinese cultural context that integrates cognitive empathy into LLMs. |
| Outcome: | The proposed model outperforms existing models in key evaluation metrics, particularly in empathy, comprehensibility, and professionalism. |
StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall (2026.acl-long)
Copied to clipboard
| Challenge: | Current benchmarks for memory utilization ignore this nuance, treating memory as a static repository of facts rather than a dynamic resource to be strategically deployed in character-centric dialogues. |
| Approach: | They propose a benchmark to evaluate strategic memory use in character-centric dialogues . they use a dataset of 657 instances where virtual characters must navigate heterogeneous memory pools . |
| Outcome: | The proposed benchmarks show that all models perform well at distinguishing between required and irrelevant memories, but struggle once supportive memories are introduced into the decision process. |
WikiFactDiff: A Large, Realistic, and Temporally Adaptable Dataset for Atomic Factual Knowledge Update in Causal Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Factual update is a task of inserting, replacing, or removing facts in large language models. |
| Approach: | They present a dataset that describes the evolution of factual knowledge between two dates as a collection of simple facts divided into three categories: new, obsolete, and static. |
| Outcome: | The proposed dataset compares the state of the Wikidata knowledge base at 4 January 2021 and 27 February 2023. |
Detecting AI-Generated Video: A Vision–Language Dual-View Survey (2026.findings-acl)
Copied to clipboard
| Challenge: | realism of AI-generated Videos (AIGC-V) rendering artifact-centric detection insufficient, authors argue . a vision–language dual-view taxonomy is proposed to systematize this rapidly evolving field . |
| Approach: | They propose a Vision–Language Dual-View taxonomy to systematize AIGC-V detection . they propose realism of AI-generated Videos is rendering traditional inspection insufficient . |
| Outcome: | The proposed model aims to show that the existing methods are consistent with real-world facts. |
DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for cultural awareness and alignment are lacking . Existing datasets for culture specific items (CSIs) focus primarily on concepts at the regional level and may contain false positives. |
| Approach: | They propose a new CSI dataset for Indian culture that measures cultural competence . they use a CSI created by LLM as Judge and human evaluations from diverse regions . |
| Outcome: | The proposed model shows that it is capable of generating culturally relevant adaptations across multiple cultural facets. |
seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | **seqBench** allows systematic variation of several key complexity dimensions. |
| Approach: | They introduce a parametrized benchmark for probing sequential reasoning limits in Large Language Models through precise, multi-dimensional control over several key complexity dimensions. |
| Outcome: | The framework allows systematic variation of logical depth, backtracking requirements and noise ratio on state-of-the-art LLMs. |
VisAidMath: Benchmarking Visual-Aided Mathematical Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing Large Multi-modal Models lack a robust visual processing capability that is often masked by evaluation metrics that prioritize final-answer accuracy. |
| Approach: | They propose a three-layer evaluation framework that scrutinizes the generation of valid visual aids and the soundness of subsequent reasoning steps. |
| Outcome: | The proposed framework examines the generation of valid visual aids and the soundness of subsequent reasoning steps on state-of-the-art models. |
ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work in language modeling has led to effective SLMs with impressive performance levels across various benchmarks. |
| Approach: | They propose a benchmark that introduces process-level evaluation for commonsense reasoning tasks. |
| Outcome: | The proposed benchmarks show that large language models provide correct answers despite flawed reasoning processes in a substantial portion of cases. |
A Survey of Deep Learning for Geometry Problem Solving (2026.acl-long)
Copied to clipboard
| Challenge: | Recent surge in deep learning technologies has significantly accelerated research in this area. |
| Approach: | They propose a comprehensive summary of the relevant tasks in geometry problem solving and a review of related deep learning methods. |
| Outcome: | The proposed method is based on a systematic review of related methods and evaluation metrics and methods. |
Structure Guided Retrieval-Augmented Generation for Factual Queries (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for RAG produce factually incorrect outputs, resulting in incorrect answers. |
| Approach: | They propose a novel problem that explicitly incorporates structural information into RAG for factual questions to satisfy all query conditions. |
| Outcome: | The proposed method significantly outperforms baselines on ERQA while maintaining reasonable computational overhead. |
ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agents (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies on proactive dialogue models focus on domain-specific or task-oriented scenarios, which leads to fragmented evaluations and limits the comprehensive exploration of models’ proactive dialogue abilities. |
| Approach: | They propose a framework for evaluating proactive dialogue capabilities of large language models that decomposes proactive dialogue into target planning and dialogue guidance, establishing evaluation metrics across various domains. |
| Outcome: | The proposed framework decomposes proactive dialogue into target planning and dialogue guidance, establishing evaluation metrics across various domains, and enables automatic generation of diverse and challenging evaluation data. |
Evaluating Scene-based In-Situ Item Labeling for Immersive Conversational Recommendation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for evaluating item labels fail to leverage scenario-specific information modalities, present redundant information that is visually inferable, and lack latent awareness of users' information needs. |
| Approach: | They propose a principled categorization of information needs into explicit intent satisfaction and proactive information needs and define evaluation metrics for item label selection. |
| Outcome: | The proposed evaluation framework is based on IR-, LLM-, and VLM-based methods across fashion, movie recommendation, and retail shopping scenarios. |
CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for radiology report generation focus on lexical overlap and entity matching. |
| Approach: | They propose a benchmark to evaluate the fine-grained factual consistency of CT reports . they use a question-answering process to query a machine-generated report . |
| Outcome: | The proposed benchmark evaluates the fine-grained factual consistency of CT reports . it correlates better with expert clinical assessment and is more sensitive to errors . |
ParaSuite: Boosting LLM Reasoning via Paradox Resolution (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for paradox research focus on checking basic logical consistency and not reflective reasoning. |
| Approach: | They propose a pipeline dedicated to paradox research that automates data synthesis, evaluation, and training. |
| Outcome: | The proposed pipeline improves paradoxical and general STEM reasoning. |
Legal Judgment Prediction: A Reflection on the State of the Art (2026.acl-long)
Copied to clipboard
| Challenge: | Legal Judgment Prediction (LJP) involves predicting judgment outcomes based on fact descriptions of cases. |
| Approach: | They propose to use argument trees to build automated legal judgment prediction systems that are trustworthy and can be used to predict cases. |
| Outcome: | The proposed model outperforms competitors on standard evaluation datasets and enables pluralistic values to be naturally expressed. |
JARVIS or Ultron? A Survey on the Safety and Security Threats of Computer-Using Agents (2026.acl-long)
Copied to clipboard
Ada Chen, Yongjiang Wu, Junyuan Zhang, Jingyu Xiao, Shu Yang, Jen-tse Huang, Kun Wang, Wenxuan Wang, Shuai Wang
| Challenge: | Recent advances in computer-using agents have created new safety and security risks . despite the impressive capabilities of CUAs, there are still significant security risks. |
| Approach: | They propose a systematization of knowledge on the safety and security threats of Computer-Using Agents. |
| Outcome: | The proposed framework provides a framework for assessing the safety and security risks of computer-using agents. |
From Words to Pixels: A Comprehensive Survey on Large Language Models in Visual Segmentation (2026.acl-long)
Copied to clipboard
| Challenge: | Visual segmentation with instruction has been a challenging task for many years . large language models and large multimodal models have spurred a new wave of research . |
| Approach: | They review recent works in LLM-based visual segmentation and analyze their architectural innovations, training strategies, and benchmark performance. |
| Outcome: | The present study reviews the most recent works in LLM-driven visual segmentation . it identifies key challenges and promising future directions . |